External voice agents in NethVoice: passing the call back to the PBX with the conversation

Hi all,

Following the thread on using Hermes agents from other NS8 apps, this one is about the phone line only: a voice agent that runs outside NethVoice, answers calls the PBX sends to it, and passes the caller back to a person of the PBX when needed. We would like to contribute this to the voice agents work, and we want to agree on the shape here before opening a pull request.

@Stll0 @mrchiao, you are working on the voice agents, so your opinion matters most here.

What we tested

On a lab NethServer 8 with NethVoice 1.8.4, our voice agent runs in a container on the same server and is reached through a SIP trunk. For the demo we invented a property management office:

  • The agent is a custom extension, so the admin can pick it as a destination. A time condition sends calls to the agent for two hours and to the operators’ ring group outside that window.
  • The agent answers from the office’s documents and collects the caller’s name, building, request and urgency.
  • When the caller asks for a person, the agent calls the ring group. The colleague’s phone shows the caller’s number and a short summary as the caller name. The caller stays with the agent while it rings: if nobody answers or the colleague declines, the agent goes on with the caller; if someone answers, caller and colleague talk and the agent leaves.

This works without any change to NethVoice, but it has two limits. The audio of the whole call keeps going through the agent, so the call uses two trunk channels until the end. And the PBX sees the hand-over as a new call coming from the agent, so its fallback, voicemail and call reports don’t follow the caller.

We also tried a SIP REFER towards a PBX number, with the summary in SIP headers (Asterisk exposes them as SIPREFERTOHDR). The audio then stays inside the PBX, but if nobody answers, the call is dropped.

What we saw in the voice agents pull requests

ns8-nethvoice#1071 and #1120 add agent types: Receptionist (Basic Voice), the private consultation before a transfer, and CleverAI for agents that run outside NethVoice. A CleverAI agent is a destination in VisualPlan with a fallback output, and the PBX sends it four headers (X-OS-Caller, X-OS-Caller-Name, X-OS-DID, X-OS-Extension). It works only through the OpenAI SIP trunk, and the docs say: “Transfers from CleverAI back into NethVoice are deferred; no REFER or transfer API is added.”

That deferred part is what we have been testing, so we would like to help with it.

What we propose

  1. Any SIP trunk for an external agent. The same type as CleverAI (VisualPlan destination, the four headers, fallback output), but with a plain SIP trunk to the agent instead of OpenAI SIP. Would you extend the CleverAI type, or add a separate “external SIP agent” type?
  2. Passing the call back. The external agent asks the PBX to transfer the caller to one of the destinations that agent is allowed to reach, with the same routing permissions as the Receptionist (extensions, queues, IVRs). NethVoice checks the permission. If nobody answers, the agent’s VisualPlan fallback runs instead of hanging up. Would you rather receive a SIP REFER, or a call to an API?
  3. The conversation goes with the call. One short plain-text header with the summary (for example X-OS-Reason). NethVoice keeps it on the transferred call and shows it to the colleague (the caller name on the phone, the CTI). When the destination is a Receptionist, it uses the header as the reason for the consultation, so the caller does not repeat themselves. Which name and length would you accept?
  4. Where to land it. Should the pull request target the voice agents branch, or wait until #1071 is merged?

We will write the code, the tests and the docs, and test every case on our lab: answered, no answer, declined, caller hangs up. Happy to open a draft pull request as soon as we agree on the shape.

Thanks!
Michele Vitiello Bonaventura • CallistoAI

Here is a short video of the hand-over on our lab. NethVoice is 1.8.4, the current stable release, with no change to NethVoice itself.

The call is simulated: an extension dials out to the agent’s number through the SIP trunk, as an outside caller would. When the caller asks for a person, the agent calls the operators back on the PBX, and the caller stays with the agent while it rings. In the video it happens twice. The first time extension 201 is on Do Not Disturb, so nobody takes the call and the agent goes on with the caller. Then 201 switches to Available, the agent tries again, 201 rings and answers, and the agent leaves.

This recording is from before the summary on the display: here 201 only sees the number the call comes from.