Voice AI Latency: Causes, Measurement, and Caller Impact

Voice AI latency is the pause between the moment a caller stops speaking and the moment the voice agent starts answering. Retell AI measures it end to end, from the end of the caller's speech to the start of the agent's speech, and reports it for every call. Callers do not see a dashboard. They hear that pause on every turn, and it shapes whether the call feels like a conversation.

This page explains why the pause matters, where it comes from, how to measure it properly, and which changes actually shorten it.

Why does a one-second pause matter on a phone call?

Human conversation runs on very short gaps. A 2009 study of turn-taking across 10 languages measured a mean gap of about 208 milliseconds between a question and its reply. The same paper notes earlier research on English showing that replies which fail to answer the question are often delayed by up to one second.

That second finding is the important one for voice agents. Listeners read a long pause as a sign that something is wrong. On a phone call, a long gap also invites the caller to fill it, which often means both sides start talking at once. The glossary entry on latency in voice systems covers the basic definition.

Judge latency at P90 and P99, not the average. A median that looks fine can hide the slow turns callers actually notice. Retell reports P50, P90, P95, and P99 for every call.

Where does the delay come from in a voice agent?

Each turn passes through several steps, and each one adds time. Retell's per-call latency breakdown separates them:

• Transcription (ASR): turning the caller's audio into text, including the endpointing window, a short silence the system waits through to confirm the caller has finished.
• Language model (LLM): the time from sending the conversation to the model until the first speakable words come back. With a custom LLM, this includes the round trip to your server.
• Knowledge base: retrieving relevant content, when the agent uses one.
• Text-to-speech (TTS): the time until the first byte of audio is ready.
• Speech-to-speech: agents on a speech-to-speech model report one combined figure instead of separate transcription, model, and voice steps.

Two more sources sit outside the models. Response Wait time is a deliberate pause you can add before every reply, up to 5.5 seconds. Network distance matters too: calling across continents adds delay, so use a phone number in the same region as your callers. How these steps fit together on a live call is covered in how a voice agent works during a call.

How do you estimate a latency budget?

Add up the steps to see which one dominates. Because Retell streams each stage into the next, the real figure can be lower than the simple sum, so use the sum as a budget and confirm it against measured calls.

Voice agent latency budget: Turn latency budget = Transcription + LLM to first words + TTS to first audio + Response wait time

300 ms transcription + 400 ms LLM + 150 ms TTS + 0 ms added wait = an 850 ms budget per turn. Add a 1 s response wait for slow speakers and the budget becomes 1,850 ms, above the 1.5 s estimate Retell suggests staying under.

Stages stream into each other, so treat the sum as a ceiling and check it against measured end-to-end latency.

Retell shows an Estimated Latency figure on the agent page before you go live. Settings that add delay carry a turtle icon, and any response wait you add appears as its own row. Retell's guidance is to keep estimated latency under 1.5 seconds and turn off any turtle-marked feature you do not need.

How is voice agent latency measured?

Retell records end-to-end latency for each call in Call History and returns the full breakdown through the Get Call API after the call ends. Three details matter when you read it:

• Only speaking turns count. Turns where the agent only runs a tool without saying anything are excluded, so tool execution time does not inflate the number.
• The network leg to the caller is not included. The figure stops at Retell's server, not the caller's handset.
• It is reported as percentiles. Each component comes with P50, P90, P95, and P99 values plus min and max, all in milliseconds.

Percentiles matter because averages hide the bad turns. In Retell's own example response, a call has a median end-to-end latency of 800 milliseconds but a P99 of 2,500 milliseconds. The caller remembers the 2.5-second gap, not the median.

At fleet level, the analytics dashboard charts end-to-end latency (average, median, P90, min, max) over time, and AI QA can score latency against a threshold you set as part of your resolution criteria. Comparing platforms on the same test is a separate exercise, covered in voice AI latency benchmarks.

What makes latency worse, and what fixes it?

Start with the per-call breakdown and fix the largest component. Retell's troubleshooting guide puts the biggest levers first:

1. Model choice. Smaller models respond faster. Larger reasoning models trade speed for capability. Re-test the agent's behavior after switching, because the model change affects answers, not just speed.
2. Prompt length. Every token is paid on every turn. Prompts beyond roughly 8,000 tokens get noticeably slower. Move rarely used detail into the knowledge base.
3. Fast tier. Routes model calls through dedicated, high-priority infrastructure for more consistent response times. It costs 1.5 times the standard model rate, so check the Retell pricing page before turning it on.
4. Transcription settings. "Optimize for speed" responds faster than "Optimize for accuracy." Boosted keywords can also add transcription time.
5. Voice settings. Some options, such as emotion, can slow speech generation, and speed varies by voice provider.

If audio is choppy rather than slow, the cause is usually network jitter. Retell's docs suggest a ping test to its API (a round trip consistently above 300 milliseconds can cause spikes) and, for a specific call, opening its packet capture in Wireshark. The full sequence is in Retell's guide to troubleshooting high latency.

• Open the per-call latency breakdown and find the largest component.
• Try a smaller or faster model, then re-test the agent's answers.
• Trim the prompt and move rarely used detail into the knowledge base.
• Turn off turtle-marked settings and Agent Handbook presets you do not need.
• Use speed mode for transcription except on steps that collect numbers or dates.
• Check the response wait time before blaming the model.
• Use a phone number in the same region as your callers.

What should the agent do while a tool call is running?

A lookup in your CRM or scheduling system can take longer than any model step. The fix is to fill the silence, not to hide it. Retell's "speak during execution" option has the agent say a short line such as "One moment, let me check that" while the tool runs, and the docs recommend it for any tool that takes over a second. Track the slow ones with the average custom tool latency in AI QA, or set an alert on custom function latency.

Is faster always better?

No. The fastest settings can hurt the call. A short endpointing window cuts off callers who pause mid-sentence. And accuracy mode, not speed mode, is the documented choice for steps that collect account numbers or dates, such as on healthcare intake calls that ask for a date of birth. The right setup is usually fast by default, with accuracy mode or a longer wait only on the steps that need it. The details of that tradeoff are in how voice agents handle interruptions and turn-taking.

The quickest way to find your own numbers is to build an agent, place a few test calls, and read the breakdown. Start free in the Retell dashboard and check the estimated latency before your first call.