How AI Voice Agents Work on a Phone Call

On a phone call, an AI voice agent works in three phases. Before the caller hears anything, it loads context about who is calling. During the call, it repeats one loop on every turn: transcribe the caller, decide the reply, take any action, and speak. After hang-up, it writes a structured record for your team and systems.

This page follows one inbound call from ring to record: a customer phoning a home-services customer support line to move a repair visit. Each stage names the Retell AI setting that controls it. For the definition and the list of parts, see what an AI voice agent is. For the wider topic, start with how voice AI works.

• Before pickup: What the agent does: Identifies the caller and loads context. Retell setting to check: Inbound webhook, dynamic variables, contact memory.
• Greeting: What the agent does: Decides who speaks first and what it says. Retell setting to check: Welcome Message, Pause Before Speaking.
• Caller talks: What the agent does: Transcribes speech and judges when the turn ends. Retell setting to check: Denoising Mode, Transcription Mode, Interruption Sensitivity.
• Reply: What the agent does: Picks the answer, looks things up, speaks. Retell setting to check: Prompt or flow, knowledge base, functions, voice.
• Handoff or end: What the agent does: Transfers to a person or ends the call. Retell setting to check: Transfer Call, End Call, End Call on Silence.
• After hang-up: What the agent does: Writes the record and sends webhooks. Retell setting to check: Post call extraction, webhooks, Call History.

What happens before the caller hears the agent?

The customer dials the company's number, which has an agent bound to it as the inbound agent. It can be a number bought through Retell or an existing number imported or routed in over SIP.

If the number has an inbound webhook, Retell sends your server the caller's number and the dialed number before the call connects. Your server can reply with the agent to use, with dynamic variables such as the customer's name and booked visit, or with an instruction to reject the call. If the agent has Use contact memory turned on, it also receives a short brief from this caller's earlier calls and texts, matched by phone number.

Then the greeting. The Welcome Message setting decides who talks first. With "AI speaks first", the agent opens with a fixed line, which can include variables ("Hi Dana, calling about Thursday's repair visit?"), or one it writes from your prompt. Pause Before Speaking stops it from talking over someone still picking up.

How does the agent hear the caller on a phone line?

While the customer talks, speech recognition turns their audio into text as it arrives. On a phone line, three settings do most of the work:

• Denoising Mode. Remove noise is the default. Remove noise + background speech also filters other voices, such as a TV, but can suppress a quiet caller, so test it first.
• Transcription Mode. Optimize for speed is the default. Optimize for accuracy helps when the caller reads out an account number or a date. In a conversation flow, one node can use accuracy mode while the rest of the call stays on speed.
• Boosted Keywords and Vocabulary Specialization. Both bias recognition toward names and specialized terms, such as part names.

If the customer says "hold on, let me find the invoice", the agent can be prompted to stay silent instead of filling the pause.

How does the agent know the caller has finished?

A pause is not always the end of a turn. Retell detects when speech is complete, and the agent holds off longer when it senses an unfinished thought. Response Wait Time adds a minimum wait after the caller stops, for people who speak slowly.

Interruption Sensitivity sets how easily the caller can cut in while the agent is speaking, and backchanneling adds short sounds like "uh-huh" while the caller talks. If the caller goes quiet, reminders prompt them, and End Call on Silence closes the line after a set time. The tuning detail is in turn-taking and interruptions.

How does the agent choose its reply on each turn?

When the turn ends, the language model gets the transcript so far, your prompt, and any variables loaded before pickup. If a knowledge base is attached, Retell first condenses the recent conversation into a search query, pulls the most relevant passages from your documents, and passes them to the model. This happens before every response, with no change to the prompt.

A single prompt, Retell's recommended starting point, describes the whole task. A conversation flow splits the call into nodes for calls that must follow a fixed order. Guardrails limit what the agent may say.

In the repair call, the model reads "can we push it to next week?" and concludes it needs the open slots before it can answer.

What happens when the agent looks something up mid-call?

Instead of speaking, the model can call a function: it decides a function is needed, pulls the parameters from the conversation, Retell runs the function, and the result feeds the next reply.

Here, an integration tool checks availability on the company's Cal.com calendar and reschedules the booking. Integration tools work with providers you connect once, such as HubSpot, Salesforce, Zendesk, or Cal.com. Custom functions call your own APIs, such as a job-management system.

While a lookup runs, the agent can say a line like "Let me look that up for you" so the line does not go quiet. With a custom function, if the customer speaks during the lookup, the agent answers them and the request still runs to completion. Once the visit is moved, a Send SMS function can text the new date to the caller, on numbers with SMS enabled.

How does the reply reach the caller?

Text-to-speech turns the reply into audio in the voice you chose: one from Retell's library, a cloned voice, or a voice from your own provider account. Voice Speed runs from 0.5x to 2.0x, and an option lets the agent shift its pace toward the caller's during the call. If the voice provider fails mid-call, a fallback voice takes over so the call continues instead of dropping. For Retell's platform voices, that fallback is automatic.

Every turn runs the whole loop again. Listen, decide, act, and speak repeat each time the caller talks, so a slow lookup or an overloaded prompt is heard as silence. Call History shows each call's end-to-end latency, and the Get Call API breaks it down by component.

Every stage adds time between the caller's last word and the agent's first. Where voice AI latency comes from covers how to measure it.

When does the call go to a person or end?

If the customer asks for a person, or the request falls outside what the agent should handle, the Transfer Call function hands off the live call. A cold transfer connects the caller and drops the agent. A warm transfer keeps the agent on the line to wait for a person, brief them privately, or introduce the caller. An agentic warm transfer has a second AI agent talk to the destination first and decide whether to connect. The trade-offs are in how AI-to-human handoffs work.

A supervisor in Live Monitoring can also listen silently or take over. Otherwise, the call ends when the agent finishes the task and uses its End Call function, when the caller hangs up, after a stretch of silence, or at the maximum call duration you set.

What changes when the agent places the call?

An outbound call runs the same loop with a different start. The call is created from the dashboard, the Create Phone Call API, or a batch call, from a number with an outbound agent bound to it. Context arrives in the request as dynamic variables (one CSV column per variable for a batch).

A person may not answer. During the first 3 minutes after the call connects, voicemail detection listens for a greeting, and the agent then hangs up or leaves a message that can include the customer's name. IVR Hangup ends calls that reach a phone menu, while the Press Digits function lets the agent work through one. On inbound calls these settings have no effect, because a person called you.

What record does the call leave behind?

When the call ends, Retell sends a call_ended webhook with the call's details, including why it ended (for example user_hangup, call_transfer, or voicemail_reached). Post call extraction then reads the transcript with an LLM, and a call_analyzed webhook delivers the results.

Every call gets a summary, the caller's sentiment, whether the call met your success criteria, and whether it reached voicemail. Custom fields add what your team tracks, such as a yes/no for "visit rescheduled" or a selector for job type, and can sync to your CRM. This is post-call analysis.

Call History keeps the recording, the transcript with each tool call and result, knowledge base retrievals, the variables used, cost, and latency. With Save conversation to memory on, the contact's memory is updated so the next call picks up where this one ended. AI QA can then score a sample of calls against rules you configure.

Before a pilot, put each of these moments on a test call:

• The caller talks over the greeting.
• A noisy line or a caller on speakerphone.
• An account number or a date read out loud.
• A lookup that fails or times out.
• A transfer that nobody answers.
• An outbound call that reaches voicemail.

The quickest way to see how each stage holds up is a pilot on your own call types, with your own callers. Talk to Retell about a pilot on real calls.