Connecting an AI Voice Agent to Your Phone System
A voice agent answers calls, understands what the caller wants and replies with a synthetic voice. Under the hood it is a very different system from a chat or email agent: response time is measured in fractions of a second, audio arrives in telephone quality, and the way it plugs into your existing phone system determines how much needs to change. This guide covers the building blocks, the three common ways to connect, and the legal points to settle before the first test call.
What is an AI phone agent and what is it used for?
An AI phone agent is software that holds a phone conversation on its own: it listens, turns speech into text, decides on a reply with a language model, looks things up in business systems when needed, and speaks the answer. It earns its place where many similar calls come in, or where calls outside office hours go unanswered.
Common jobs in a B2B setting:
- Call reception and routing: ask what the caller needs and connect them to the right team, instead of a keypad menu.
- Status and routine information: order status, opening hours, the state of a service ticket, provided the agent has read access to the relevant system.
- Fault and service reports: capture device, serial number, symptoms and a callback number in a structured way and open a ticket.
- Appointment booking: offer free slots from a calendar and book one.
- Out-of-hours answering: take the request, leave a summary as an email or ticket, and trigger a callback.
The text and ticket version of this role is covered in the guide to AI customer service agents. A phone agent does the same job for the business, but on a completely different technical foundation: real-time audio, telephony signalling and replies fast enough to feel like a conversation.
How does an AI voice agent work under the hood?
A voice agent has five parts: the telephony connection, a media bridge, speech-to-text (STT), a language model with tools, and text-to-speech (TTS). Telephony delivers the call, the media bridge streams the audio in real time, and the STT → LLM → TTS chain produces each reply.
Telephony layer: SIP signalling
In IP telephony, signalling is usually handled by SIP. RFC 3261 defines SIP as an application-layer signalling protocol for creating, modifying and terminating sessions. SIP does not carry the audio itself; that travels alongside as a media stream (RTP). In practice, some component has to answer the SIP call and pick up the media.
Media bridge: from call to audio stream
The media bridge links the telephone world with your agent logic. Three examples show the range of options:
- Cloud telephony with WebSocket streaming: Twilio's Media Streams stream raw audio from live calls over WebSockets to your application in near real time, and bidirectional streams let the application play audio back into the call. Per the message reference, payloads are
audio/x-mulawat 8,000 Hz — classic telephone quality. - Your own Asterisk-based PBX: the Asterisk AudioSocket channel is a simple TCP protocol that carries real-time audio as 16-bit PCM and passes DTMF digits.
- SIP-native agent platforms: LiveKit SIP accepts calls on inbound trunks and uses dispatch rules to route them into rooms where the agent runs. The LiveKit server is Apache 2.0-licensed according to its GitHub repository and can be self-hosted, in which case the SIP service is deployed separately.
The STT → LLM → TTS latency chain
Every reply passes several stages. The audio has to be recognised, the system has to decide the caller has finished, the model generates text (sometimes after a tool call), and speech synthesis has to produce the first syllables. Each stage adds waiting time, and so does every network hop between services.
What makes the difference in practice:
- Streaming at every stage: speech recognition emits partial results, the model streams tokens, and synthesis starts on the first sentence before the last one exists.
- Good end-of-turn detection: waiting too long for silence feels sluggish; jumping in too early talks over the caller.
- Barge-in: when the caller interrupts, playback must stop at once.
- Short network paths: run the telephony entry point, the model and the speech services in the same region where possible.
- Covering tool calls: if a lookup takes a moment, the agent says so ("Let me check that") rather than going silent.
General levers for response time and cost — model choice, caching, context size — are covered in the guide to AI agent cost and latency. On the phone there is an extra twist: you cannot hide the wait behind a loading spinner, because the caller just hears silence.
The alternative to a three-stage chain is a speech-to-speech model that works on audio directly. OpenAI offers SIP access to its Realtime API: according to a Twilio walkthrough, the SIP trunk points at a project-specific SIP address at OpenAI, a webhook announces each call with the realtime.call.incoming event, and your application answers it with an accept request. That shortens the chain but ties model, voice and data processing to one vendor.
Tools and handover to people
Without access to business systems, an agent can only give general answers. Tools are defined functions — create a ticket, check a calendar, verify a customer number — that the model can call (see tool use and function calling). Equally important is the handover: a SIP transfer to an extension or hunt group when the agent is stuck or the caller asks for a person.
What are the options for connecting to a phone system?
There are three common routes: forwarding calls from your existing system to a number on the agent service, a SIP trunk between your phone system and the agent platform, or a dedicated cloud number that terminates directly at the agent service. They differ mainly in effort, control over numbers, and how cleanly calls can be handed back to staff.
| Criterion | Call forwarding | SIP trunk from your PBX | Dedicated cloud number |
|---|---|---|---|
| Change to phone system | small (forwarding rule) | moderate (trunk, routing) | none |
| Existing numbers | kept | kept | new number or porting |
| Transfer back to staff | often clumsy, via external number | direct to internal extensions | via external numbers |
| Control over routing | low | high | with the cloud provider |
| Good fit for | pilots, out-of-hours | permanent use, many extensions | new hotlines, campaigns |
Call forwarding
Your phone system forwards selected calls — after hours, when busy, or after a ring timeout — to a number that lands on the agent service. It is the quickest way to run a pilot. The downside is that the call leaves your system, and handing it back to an internal extension only works via an external number, often losing the caller's original caller ID on the way.
SIP trunk between PBX and agent platform
Your phone system gets an extra SIP connection to the agent platform and treats the agent as an extension or a routing target. Calls can be allocated precisely, and the agent can transfer back to internal extensions. This needs access to the PBX configuration, matching codecs and firewall rules. The details — registration, authentication, codecs and transfers — are covered in a dedicated guide in this section.
Dedicated cloud number
The agent gets a number directly from a cloud telephony provider, and the call never touches your own PBX. This suits new hotlines or temporary numbers. Moving an existing main number to this route requires porting, and transfers to staff go via external numbers.
What do you need before you start?
You need an IP-capable phone system or a cloud telephony provider, a clear list of call reasons, access to the systems the agent should use, and an agreed fallback for handovers. On top of that come data protection documents for every service in the chain.
Technology
- A phone system with SIP support, or a cloud telephony provider offering forwarding or a streaming interface.
- Network rules for SIP and media traffic; for self-hosted components, fixed IP addresses or ranges.
- APIs into the ticketing system, CRM, calendar or ERP, with a separate, restricted account for the agent.
Content
- Data on why people actually call: call logs or a simple tally over two weeks is enough.
- For each call reason: what the agent must ask, what it may answer, and when it hands over.
- A greeting that discloses the AI and, if calls are recorded, mentions recording.
Organisation
- A fallback target for handovers (team, extension, callback list) with a named owner.
- Early involvement of your data protection officer and, where one exists, the works council or employee representatives — especially if calls are recorded or staff can listen in.
How do you roll it out step by step?
Start narrow: one call reason, one connection route, a limited time window. Add further call reasons and move to a permanent SIP trunk only once recognition, response time and handover are stable in the pilot.
- Pick a call reason: frequent and well defined, such as fault reports or out-of-hours calls.
- Design the conversation: greeting with AI disclosure, required details, permitted answers, exit and handover rules.
- Choose the connection: forwarding or a separate test number for the pilot, a SIP trunk for permanent use.
- Choose the stack: a three-stage chain with interchangeable services, or a speech-to-speech model; check regions and data processing.
- Connect tools: read-only first, write access only after testing — for example, creating a ticket rather than editing a record.
- Test internally: real handsets, mobile networks, speakerphone, accents, background noise and interruptions.
- Go live in a limited way: at off-peak times or for a share of calls, reviewing transcripts daily where you are allowed to keep them.
- Expand: more call reasons, more tools, a switch to a SIP trunk.
In our own operations, the handover to people needed more coordination than the agent itself: who picks up transferred calls, and how does the agent's summary reach them?
What are the limits and legal risks?
The technical limits are response time, recognition on poor lines and model errors. On the legal side, EU deployments need to address three things: recording rules, GDPR information duties and the AI Act's transparency obligation. What follows is orientation, not legal advice; recording law in particular varies by country.
Technical limits
- Audio quality: narrowband telephone audio (8,000 Hz with Twilio Media Streams), mobile networks and speakerphones make recognition harder, especially for names, serial numbers and part numbers. Plan for spelling and read-back.
- Response time: each stage of the chain and each tool call lengthens the pause; long pauses cause people to talk over the agent or hang up.
- Wrong answers: language models can produce plausible but false answers, known as AI hallucination. Binding information — prices, delivery dates, commitments — should only come from system data, never from the model itself.
- Scope: complaints, escalations and emotionally charged calls belong with a person, quickly.
Recording
Rules on recording calls differ between jurisdictions. In Germany, for example, Section 201 of the Criminal Code makes it an offence to record another person's privately spoken words without authorisation. A voice agent necessarily processes the conversation; whether and for how long audio or transcripts are stored is a deliberate choice. Common practice is a notice at the start and consent before recording — and not storing anything you do not need.
GDPR information duties
Article 13 GDPR requires that people are informed at the time their data is collected about, among other things, the purposes and legal basis of the processing. On the phone this is usually a short notice in the greeting plus a pointer to the privacy notice. Every service in the chain — telephony, speech recognition, model, speech synthesis — needs a data processing review and, where relevant, a check of transfers outside the EU. With that groundwork, a voice agent can be operated in line with the GDPR.
AI Act transparency
Article 50 of the EU AI Act requires that systems interacting directly with people are designed so that those people are informed they are interacting with an AI system, unless that is obvious. According to that source, the obligation applies from 2 August 2026. One sentence in the greeting ("You are speaking with a digital assistant") is the simplest way to meet it. Recording, consent and disclosure wording are covered in more depth in a dedicated guide in this section.
How do you get started?
Begin by analysing your incoming calls and choosing one frequent call reason with clear rules. In parallel, find out which connection route your phone system supports and bring in data protection early. A time-boxed pilot via call forwarding quickly shows whether recognition and response time work for your callers.
A short starting checklist:
- Which three call reasons account for most of your calls?
- Does your phone system support SIP trunks, and who can change its configuration?
- Which systems must the agent read, and which may it write to?
- Where do handed-over calls go, and who owns them?
- Will you store recordings or transcripts — and if so, for how long and on what legal basis?
- In which region do telephony, model and speech services run?
How voice agents fit alongside other business agents, such as those for quotations or technical service, is covered in the overview of AI agents for businesses. Technical service has its own page on the technical service agent, which can pick up fault reports from the phone channel. If you want telephony, model and speech services running in your own infrastructure, start with the basics of sovereign AI.
Frequently asked questions
Do I need to replace my phone system to use an AI phone agent?
Usually not. Most IP-based phone systems can hand calls to an external service by forwarding or through an additional SIP trunk.
Which audio format does the agent receive from the phone network?
Often narrowband telephone audio. Twilio Media Streams, for example, deliver mu-law audio at 8,000 Hz, which is why speech recognition should be tested on real calls rather than studio recordings.
Do callers in the EU have to be told they are talking to an AI?
Yes. Article 50 of the EU AI Act requires that people are informed when they interact with an AI system unless it is obvious. A single sentence in the greeting is the simplest way to comply.
Can the voice agent transfer a call to a person?
It should. With a SIP trunk the agent can transfer to internal extensions; with forwarding or a cloud number, transfers usually go via an external number.
Sources
- RFC 3261: SIP: Session Initiation Protocol rfc-editor.org
- Twilio Docs: Media Streams twilio.com
- Twilio Docs: Media Streams WebSocket Messages twilio.com
- Twilio Blog: Connect the OpenAI Realtime SIP Connector with Elastic SIP Trunking twilio.com
- LiveKit Docs: SIP docs.livekit.io
- LiveKit server on GitHub (licence) github.com
- Asterisk Docs: AudioSocket docs.asterisk.org
- EU AI Act, Article 50: Transparency obligations artificialintelligenceact.eu
- GDPR Article 13: Information to be provided at collection dsgvo-gesetz.de
- German Criminal Code, Section 201: Violation of the privacy of the spoken word gesetze-im-internet.de