Conversational Turn-Taking and Interruptions in Voice AI

Conversational turn-taking is the set of split-second decisions that settle who speaks when. For a voice agent it comes down to three calls made on every turn: has the caller finished, is the caller trying to cut in, and should the agent acknowledge them without taking the floor. Get those right and the call feels like a conversation. Get them wrong and the agent talks over people or leaves dead air.

This page explains how each decision works, which settings control it in Retell AI, and how to tune them for your callers.

Why is turn-taking hard for a voice agent?

People are very fast at it. A 2009 study of conversations in 10 languages found that speakers everywhere avoid overlapping talk and keep the silence between turns short. The mean gap before a reply across the whole dataset was about 208 milliseconds, and every language's average fell within roughly 250 milliseconds of that.

A voice agent has to make the same call from audio alone, with no eye contact or body language. Silence is an unreliable signal. Callers pause to find a card, read out a number in chunks, or think mid-sentence. If the agent treats every pause as the end of a turn, it interrupts. If it waits too long to be safe, it feels slow. That tradeoff runs through every setting below. A shorter definition of the concept is in the glossary entry on turn-taking endpoints.

How does an agent know the caller has finished speaking?

The first signal is endpointing: after the caller stops, the system waits a short silence window to confirm they are done before it responds. A longer window gives more accurate transcripts but adds directly to the delay the caller hears.

Retell exposes this through Transcription Mode. "Optimize for speed" is the default. "Optimize for accuracy" helps when collecting account numbers or dates, or when transcripts cut off before the caller finishes. "Custom Settings" lets you pick a speech recognition provider and set its endpointing directly. For Deepgram, the dashboard slider runs from 100 to 10,000 milliseconds, with 1,000 milliseconds as the default (as of October 2026). In a conversation flow, you can switch to accuracy mode only on the node that collects a date of birth and keep speed mode everywhere else.

The second control is Response Wait time, a deliberate minimum pause before the agent replies. It defaults to no added wait and goes up to 5.5 seconds. Retell's orchestration layer also holds off longer when it detects the caller has not finished their thought, and an optional "dynamically adjust" setting gives slower speakers more patient timing during the call.

Response wait time from the API value: Added wait (seconds) = 1 + 0.5 × ((0.9 − responsiveness) ÷ 0.1), for responsiveness from 0 to 0.9

Responsiveness 0.7: 1 + 0.5 × ((0.9 − 0.7) ÷ 0.1) = 1 + 0.5 × 2 = 2.0 seconds before each reply. On a call with 15 agent turns, that adds 30 seconds of deliberate silence across the call.

The full wait is added to every turn, so raise it only on the steps where callers pause mid-thought.

Other stacks approach end-of-turn detection the same way. LiveKit Agents, for example, documents a turn detector model that "predicts end of turn from both the meaning of speech and its acoustic properties, on top of VAD."

What happens when the caller interrupts the agent?

When a caller starts talking over the agent (often called barge-in), the agent should stop and listen. Retell controls this with Interruption Sensitivity, a value from 0 to 1:

• Higher values make the agent easier to interrupt.
• Lower values require the caller to speak longer, or say more words, before the agent yields.
• At 0, the caller cannot interrupt the agent at all.

New blank voice agents created in the dashboard start at 0.9. The API defaults to 1 when the field is left out. In a conversation flow, individual nodes can override interruption sensitivity, response wait time, voice speed, and whether keypad presses can interrupt the agent. That lets a required disclosure play through while the rest of the call stays responsive. Teams running their own model through Retell's custom LLM connection can also mark a single response as not interruptible.

One caveat: a caller who keeps cutting in to ask for a person is not a tuning problem. That is a handoff signal, and the agent should act on it. See how call transfers to human teams work.

Why do background noise and side conversations cause false interruptions?

A TV, a coworker, or traffic can sound like the caller starting to speak. The agent stops mid-sentence for no reason, and the caller hears it cut itself off.

Fix it in two steps, in this order. First, set denoising. Retell's default mode removes background noise with almost no effect on transcription accuracy, but it will not remove loud background speech. A stronger mode removes background speech too, at an extra $0.005 per minute, and can suppress a quiet or distant caller, so test it on real recordings first. Second, if false interruptions continue, lower interruption sensitivity in small steps, for example from 0.9 to 0.8, and confirm callers can still cut in when they need to. Retell's guide to handling background speech and noise covers when each mode fits.

• Agent cuts in mid-sentence: Likely cause: Endpointing too short. Setting to try first: Accuracy mode or longer endpointing on that step.
• Agent stops talking for no reason: Likely cause: Background noise or speech. Setting to try first: Denoising mode, then lower interruption sensitivity.
• Agent keeps talking over the caller: Likely cause: Sensitivity set too low. Setting to try first: Raise interruption sensitivity.
• Long pause before every reply: Likely cause: Response wait time too high. Setting to try first: Lower response wait time.

What is backchanneling, and when should a voice agent use it?

Backchanneling is the small "uh-huh" or "I see" a listener says while the other person keeps talking. It signals attention without taking the turn. In Retell it is off by default. When enabled, a frequency setting from 0 (never) to 1 (whenever possible) controls how often it fires, with 0.8 as the default, and you can edit the word list for each language.

It works best on calls where the caller talks at length, such as describing a problem or giving a history. It is less useful on short, transactional calls. The tradeoffs are covered in more depth in how backchanneling improves the caller experience.

What should the agent do when the caller goes quiet?

Silence is a turn-taking problem too. Retell gives the agent three ways to handle it:

• Reminders: if the caller has been silent after the agent speaks, the agent prompts them. By default the reminder fires after 10 seconds of silence, and the agent reminds once. Both values can be changed, and setting the count to zero turns reminders off.
• Staying silent on purpose: when a caller says "hold on" or "give me a minute," you can instruct the model to output NO_RESPONSE_NEEDED, and the agent waits without speaking.
• End call on silence: ends the call automatically after a set period of no response.

How do you test and tune turn-taking?

Tune one setting at a time and listen to real calls after each change. On a customer service line where callers read out order numbers, the right answer is usually accuracy mode on that step plus a modest response wait, not a slow agent everywhere.

• Pause for three seconds mid-sentence and check the agent waits.
• Read a 10-digit number in chunks and check it is captured whole.
• Interrupt the agent mid-answer and check it stops and listens.
• Play a TV or background voices and check the agent does not stop itself.
• Say "hold on" and check the agent stays quiet until you return.
• Stay silent after a question and check the reminder fires once.

Then measure at scale. Retell's AI QA counts overlapping speech, the times the caller and agent talked at once, and scores agent naturalness, which includes turn-taking behavior. A rising overlap count after a prompt or setting change is an early warning. These checks belong in your voice agent QA scorecard.

Every extra millisecond of waiting also shows up as latency, so read this alongside how voice AI latency affects the call experience. For where turn-taking sits in the full call pipeline, see what happens inside a voice agent during a call.

The quickest way to feel these settings is to change one and call the agent yourself. Start building for free in the Retell dashboard and run a test call against your noisiest real-world scenario.