What Is an AI Voice Agent? How One Works During a Call

An AI voice agent is software that answers or places phone calls and holds the conversation itself: it listens to the caller, decides what to say or do next, takes actions in your systems, and speaks the reply out loud in real time. To the caller it is one conversation. Underneath, every turn passes through four layers: telephony, speech recognition, a language model, and speech synthesis.

This page follows a single inbound call from ring to wrap-up and shows what each layer does, where things go wrong, and which settings control it. The examples use Retell AI, which runs the telephony, speech, and model layers on one platform so a team configures the agent instead of wiring providers together.

• Telephony: What it does on the call: Connects the call by number, SIP trunk, or web. Retell setting to check: Phone numbers, custom telephony.
• Speech recognition: What it does on the call: Turns the caller's audio into text. Retell setting to check: Denoising mode, transcription mode.
• Language model: What it does on the call: Decides the reply and which tool to use. Retell setting to check: Prompt or conversation flow, model.
• Text-to-speech: What it does on the call: Speaks the reply back to the caller. Retell setting to check: Voice, voice speed, fallback voice.

What happens when a call reaches an AI voice agent?

The call arrives over telephony. With Retell, that can be a number bought from Retell, an existing number connected over SIP trunking, or a web call from a browser. Teams that want to keep their existing phone number over a SIP connection route calls in from their current carrier or contact center platform.

Once the call connects, the agent has to decide who speaks first. In Retell this is the Welcome Message setting. "User speaks first" keeps the agent silent until the caller talks. "AI speaks first" opens with either a fixed greeting or one the model writes from your prompt. A Pause Before Speaking control stops the agent from talking over someone who is still picking up the handset.

On outbound calls there is an extra step. The agent may reach a voicemail greeting or a phone menu instead of a person. Retell runs voicemail detection during the first 3 minutes after connect, and the agent can either hang up or leave a message.

How does the agent hear and understand the caller?

Automatic speech recognition (also called speech-to-text or ASR) turns the caller's audio into text as they talk. Phone audio is noisy, so two settings matter most:

• Denoising mode filters background noise before transcription. Retell's default removes noise. A stronger mode also removes background speech, which helps on a loud line but can distort a quiet caller's voice.
• Transcription mode trades speed against accuracy. "Optimize for speed" is the default. "Optimize for accuracy" helps when the agent collects account numbers or dates.

Boosted keywords and vocabulary specialization bias recognition toward names, drug names, or product terms the general model would mishear. A wrong transcript produces a wrong answer, so this layer is where many "the agent misunderstood me" complaints start.

How does the agent know when it is its turn to talk?

A pause is not the same as a finished thought. The agent has to judge whether the caller is done, still thinking, or about to interrupt. Retell handles this in its orchestration layer with endpointing, configurable interruption sensitivity, and optional backchanneling (short "uh-huh" acknowledgments while the caller speaks).

Get this wrong and the agent either cuts people off or leaves long, awkward silences. The full mechanics, including which settings to change for slow speakers or noisy lines, are in how voice agents handle interruptions and turn-taking.

How does the agent decide what to say?

A large language model reads the conversation so far plus your instructions and produces the next reply. In Retell those instructions take one of two shapes:

• Single prompt: one set of instructions covers the whole task. Retell recommends starting here for most agents, including lead qualification calls, support, and appointment booking.
• Conversation flow: the call is split into nodes with defined transitions. This suits calls that must follow a fixed order, such as patient intake or a compliance script.

Two more inputs shape the answer. A knowledge base built from your documents or website lets the agent answer from approved content instead of the model's general training. Guardrails set limits on what the agent is allowed to say.

Retell also supports speech-to-speech models, which take audio in and produce audio out without a separate text step. This page follows the separate-layer setup, where speech recognition, the model, and the voice can each be tuned on their own.

How does the agent take action during the call?

Talking is half the job. The model can also call functions, also known as tools, mid-conversation. Retell's built-in functions cover call control: end the call, transfer the call to a person or department, press digits to move through a phone menu, and send an SMS. Integration tools connect to systems such as HubSpot, Salesforce, Zendesk, and Cal.com. Custom functions call your own APIs to look up an order, check eligibility, or create a ticket.

The model decides a function is needed, pulls the required details from the conversation, and Retell runs the call and feeds the result back. If a lookup takes more than about a second, Retell's docs recommend having the agent say something like "One moment, let me check that" while it runs, so the line never goes dead.

When the agent reaches the edge of what it should handle, it hands the call to a person. The options, from a fast cold transfer to a briefed warm transfer, are covered in how call transfers work between voice agents and human teams.

How does the reply get back to the caller?

Text-to-speech converts the model's reply into audio and streams it to the caller. You pick a voice from Retell's library or bring a cloned or third-party voice, then set speaking speed between 0.5x and 2.0x. A fallback voice takes over if the primary voice provider fails mid-call, so the call continues instead of dropping.

Every layer adds time between the caller's last word and the agent's first. That gap is the difference between a call that feels like a conversation and one that feels like a walkie-talkie. Voice AI latency and how to measure it breaks down where each millisecond goes.

Every turn runs through all four layers before the caller hears a word. A slow model, a long prompt, or a slow voice provider shows up as silence on the line. Check the per-call latency breakdown to see which layer is adding the time.

What happens after the call ends?

The record of the call is where operations teams spend their time. In Retell, the end of a call triggers this sequence:

1. Retell sends a call_ended webhook.
2. An LLM reads the transcript and fills in the call's analysis fields.
3. Retell sends a call_analyzed webhook with the results.

Built-in fields include a call summary, user sentiment, whether the call was successful by your criteria, and whether it reached voicemail. You can add custom fields (yes/no, text, number, or a fixed list of categories) and sync them to your CRM. This is post-call analysis, and it is how a team learns which calls failed and why.

Two more tools close the loop. Live monitoring lets a supervisor listen to an ongoing call or take it over. AI QA scores samples of production calls for issues such as hallucinations, latency, and resolution against criteria you define.

Should you build the pipeline yourself or use a platform?

You can assemble the layers yourself. Pipecat describes itself as an "open source ecosystem for building voice and multimodal AI agents," and LiveKit Agents is another open-source option. Building means your team owns telephony connections, provider failover, turn-taking logic, and monitoring.

Hosted platforms run those layers for you. Vapi positions itself as "voice agents for builders." Retell runs telephony, speech, and model orchestration and adds testing, analytics, and QA in the same workspace. Agents can be built in the dashboard with no code or managed through the API and SDKs. Whichever route you take, judge it on real calls with your own callers, not a scripted demo. Test interruptions, transfers, and noisy lines, because that is where the layers break first.

If you are weighing a voice agent for your contact center, the fastest way to see how these layers hold up is a pilot on your own call types. Talk to Retell about a pilot on real calls.