How Voice AI Handles the Hardest Parts of a Real Call: IVR Trees, Voicemail Detection, Mid-Sentence Interruptions, and Floor-Price Escalation
Most voice AI demos look great because they avoid the four moments where real calls fall apart: the agent has to push buttons through someone else's IVR, decide in two seconds whether a human or a voicemail just answered, recover when a caller talks over it without losing the thread, and hold a price line against a customer who has figured out the rules. Each of these is a separate engineering problem with its own failure mode, and a platform that handles one well can fail badly on the others.
This piece walks through what is actually happening underneath each of those four moments — what the architecture looks like, where the tradeoffs are, and what numbers production teams are seeing in 2026. The detail matters because the gap between a demo agent and a production agent is almost entirely contained in these four behaviors.
An easy call is single-turn and bounded. A caller asks a question the agent has been trained on, the agent retrieves the answer, the call ends. A hard call breaks at least one of those assumptions: the conversation requires the agent to take an action with real consequences (book a refund, transfer money, agree to a price), the other side of the line is not the cooperative caller the demo assumed (an IVR, a voicemail box, an angry customer running an exploit), or the conversation needs to recover from something unexpected (a barge-in, a misheard intent, a third party joining the call).
What separates production-grade voice AI from a polished prototype is whether the system was designed for the second category from the start. Every behavior below — DTMF emission, asynchronous answering machine detection, semantic interruption handling, server-side guardrails — exists because someone shipped an agent without it and watched a measurable percentage of calls fail in a way the prompt could not fix.
When a voice agent calls out to an insurer, a hospital billing department, or a supplier's support line, it usually hits an interactive voice response menu before it ever reaches a human. The agent has to listen to the menu, decide which option matches the call's purpose, send the right touch-tone digit, and repeat until it gets to a person — without speaking out loud during any of it, because most IVRs ignore voice input.
The technical wrinkle that catches new builders out is that voice codecs are designed to compress human speech and routinely treat the dual-tone frequencies of a digit press as noise to be discarded. That is why an agent that "presses 2" by playing the audio of a touch tone over the call media will work intermittently, succeed in testing, and then fail on production carriers in ways that look random. The fix is to send the digit out-of-band as an RFC 4733 telephony event rather than mixing it into the audio stream — a SIP signaling message the IVR's media gateway processes directly.
There is also a timing problem that is not obvious from the documentation. Many IVRs ignore digits sent while the menu prompt is still playing, and others have a buffer that drops a sequence of digits arriving faster than a human thumb could press them. A production navigator listens for the prompt to end, waits a beat, sends the first digit, then either pauses between subsequent digits in a member-ID or account-number sequence or pauses again at the next menu level. Get this wrong and the agent will reach the wrong department roughly a third of the time on common enterprise IVRs.
The handoff at the end is the part most teams underbuild. When the IVR finally connects to a human, the agent needs to drop the navigation context — the menu options it was matching against, the state machine of "I am looking for billing" — and switch to the actual conversation it is there to have. Done well, the human hears "Hi, I'm calling on behalf of Sarah Mitchell about claim #74821," not a confused agent still trying to interpret the last menu prompt. Retell AI exposes this through a press_digit function the agent calls when a digit is required, separate from the conversation logic that runs once a human picks up.
The first three seconds after a call connects are the most consequential of any outbound conversation. If the line was answered by a person and the agent waits too long, the person says "hello?" twice and hangs up. If it was answered by a voicemail and the agent launches into its full opener, half the message ends up cut off by the beep, and the recipient hears something that sounds confused and mechanical when they play the message back later.
The legacy approach to telling these apart, called Answering Machine Detection or AMD, uses acoustic heuristics: the length of the initial silence, the duration of the greeting, the energy envelope of the audio. These methods land somewhere in the 70 to 85 percent accuracy range and produce a high enough false-positive rate that production outbound campaigns built on them waste a meaningful share of dials leaving messages on real people's lines.
The current generation of AMD reads the transcript instead of the waveform. Voicemail greetings literally identify themselves — "you've reached," "please leave a message," "after the tone" — and that language signal is far more reliable than acoustic features that look similar between a long pause and a quick "hello." Recent published research using a recurrent neural network on transcribed audio reached over 96 percent accuracy on the test set, with a path to over 98 percent when combined with a silence-detection check.
There is a mode question that matters more than the model question. Synchronous AMD waits for a verdict before connecting the call, which adds three to five seconds of dead air that real humans interpret as a robocall and hang up on. Asynchronous AMD connects the call immediately, lets a parallel classifier listen for the first second or two while the agent says something brief, and then switches behavior based on the verdict — continuing the conversation if the verdict is "human" or pivoting into a pre-scripted voicemail message if it is "machine." Asynchronous is what production deployments use. The opener is designed so it works for either audience: "Hi, this is Maya" sounds normal to a human and gives the classifier room to commit before the agent says anything irreversible.
The cost of getting this wrong scales with volume. A platform processing 40 million calls a month — Retell AI's current run rate as of January 2026 — turns a one-percent false-positive rate into 400,000 wrongly classified calls. That is the number that pushes production teams toward purpose-built infrastructure rather than the AMD signal that ships with their telephony provider.
A real conversation does not take turns cleanly. People interrupt, talk over each other, drop in "uh-huh" while the other person is still speaking, change their mind mid-sentence, and trail off without finishing the thought. Voice AI that treats every sound during its own turn as an interruption — the default behavior of basic Voice Activity Detection — sounds jittery and robotic. Voice AI that ignores all sound during its own turn cannot be interrupted at all, which feels worse the longer the agent's response runs.
The hard problem is not detecting that the caller spoke. It is deciding, within a couple of hundred milliseconds, whether what they said was a real interruption that should yield the floor or a backchannel — "right," "okay," "uh-huh," "got it" — that should be ignored so the agent can keep talking. Get this wrong in either direction and the call feels off in a way the caller will not be able to articulate but will absolutely register.
The production approach runs three signals in parallel during agent playback. A streaming voice activity detector watches for any voiced audio. A streaming transcription emits a partial transcript within roughly 100 milliseconds of the caller speaking. A semantic classifier reads that partial transcript and decides whether it carries an actionable intent ("wait, can you go back?") or just acknowledgment noise. Only actionable intents trigger barge-in, which cuts the text-to-speech stream, drops the half-spoken response, and rolls conversation state back so the language model responds to the new input rather than the prompt that produced the cut-off response.
End-to-end latency matters here because everything compounds. Natural human turn-taking sits in the 200 to 300 millisecond range. Anything under 700 milliseconds reads as conversational; above 900 milliseconds, callers notice and disengage. Retell AI's published ~600ms response latency — measured from the caller's last word to the agent's first word — comes from the recent turn-taking model update that knocked an additional 150 milliseconds off the loop, and it is what makes recovery feel natural rather than apologetic. The agent does not say "I'm sorry, I didn't catch that." It picks up where the caller redirected and keeps going.
There is one more piece that is easy to miss. When the caller does interrupt, the agent has to track what was said and what was left unsaid. If the agent was halfway through quoting a price when the caller cut in to ask about the warranty, the agent needs to remember that the price quote was incomplete and offer to come back to it. This conversation-state continuity is the difference between an agent that recovers and an agent that loses the thread.
Negotiation is where prompt-based guardrails fail most visibly. A system prompt that says "do not discount below $899" works for the 95 percent of customers who never test it. The remaining five percent — the customer who roleplays a manager call, the customer who claims a previous rep already approved a different number, the customer who simply asks the same question fifteen different ways — are exactly the ones who push hardest, and the language model will eventually concede.
The architectural fix is to take the price out of the prompt entirely and put it behind a function call. The agent can talk freely about pricing in conversation, but the moment it tries to commit to a number, the commitment goes through a propose_price function that checks the proposed value against a server-side floor tied to the SKU and the customer segment. The function rejects anything below the floor before the number is ever spoken. The floor lives in code the language model cannot see, which means it cannot be reasoned around, prompt-injected, or talked into a lower value.
The same pattern handles the related problems: refund authority caps in support, discount approval limits in retention, payment plan minimums in collections, and three-way call workflows where the agent is on the phone with a customer and an insurance carrier simultaneously. In each case, the rule is the same — every committed action runs through a function with server-side validation, and anything that fails validation either retries within the allowed range or escalates to a human. The Gladia voice AI safety research calls this pattern "hard-coded redlines — rules that live outside the model and are enforced at the orchestration level," and it is the only approach that survives adversarial users who know they are talking to AI.
There is a second-order benefit worth naming: this architecture also prevents hallucinated commitments. A voice agent without function-gated actions can confidently promise a refund the company has no record of, quote a delivery date no system actually supports, or agree to a callback that never gets scheduled. With the function layer in place, every commitment the agent makes is something a system has actually agreed to. Voice AI hallucination rates in the published research drop from a 27 percent baseline to under 5 percent once this kind of guardrail is in place, which is the difference between an agent that is useful and one that creates more cleanup work than it saves.
A voice AI built for hard calls has a defined budget for the calls it is not going to resolve. The Medical Data Systems deployment runs at a 30 percent transfer-to-human rate on inbound collections — meaning seven of every ten calls resolve without a person, and the remaining three are designed to escalate cleanly with the full transcript and the function-call history attached. Their CIO has described this publicly: the platform "now handles 100% of inbound calls with only a 30% transfer rate, scaling effortlessly, and collecting ~$280,000 per month without sacrificing patient trust."
The triggers that fire those escalations are the same four behaviors covered above, just in failure mode: the floor-price function rejected three offers in a row, the AMD classifier returned "unsure" twice on a callback, the IVR navigation failed to reach a human after five menu levels, the interruption-recovery layer detected three turn collisions in 30 seconds. Each is a defined signal, not a vibe check, and each routes the call to a human who picks up where the agent stopped rather than starting from scratch. The warm transfer is what makes the AI portion feel like a head start rather than a wasted call.
The Sunshine Loans deployment is the inverse data point: when the agent can resolve the call, it should. Their team handles more than 700,000 monthly applications with abandonment dropping from 20-30 percent down to 5-6 percent, because callers are no longer hitting voicemail or hold queues during peak volume. The 75-80 percent of calls that resolve fully without a human are calls a previous version of the same business would have either sent to voicemail or staffed up to handle.
The metrics that matter for hard-call performance do not show up in a generic dashboard. Containment rate (calls fully resolved by the agent) and transfer rate (calls that hand off to a human) are necessary but not sufficient — they tell you whether the agent finished the call, not whether it finished it correctly. The diagnostic metrics that catch the four failure modes above are different.
For IVR navigation, the right metric is not connect rate but task-completion rate by IVR target — how often the agent reached the intended department on each unique phone tree it dials, sliced by carrier. A drop in this metric on a specific number is usually a DTMF reliability issue, not a prompt issue. For voicemail detection, the right metric is the false-positive rate (humans wrongly classified as machines) tracked separately from the false-negative rate, because the cost of each direction is different and the right tuning depends on the campaign. For interruption handling, the right metric is the false-barge rate — agent responses cut short by a backchannel that should have been ignored — which is harder to surface than total interruptions but predicts caller frustration far better. For price-line and approval-cap behavior, the right metric is policy-adherence rate measured by sampling transcripts against the actual server-side rules, not by reading the agent's promises at face value.
Retell AI's post call analysis layer scores every transcript on dimensions like these rather than the two-percent sample human QA can review, which is what makes the metrics actionable at the volume serious deployments operate at. The 2025 release of Retell Assure went further, automating the QA loop itself — the platform monitors voice AI calls and surfaces improvement candidates without a human spot-checking interactions. At 40 million calls per month, that is the only way the math works.
It can navigate most standards-compliant touch-tone IVRs reliably when the digits are sent as out-of-band SIP telephony events rather than mixed into the audio. The exceptions are IVRs that require a beep before accepting input, IVRs with very short timeouts between menu and digit, and IVRs that mix DTMF with required spoken input. Production teams test against the specific phone trees they need to dial before launch and add per-IVR retry logic for the persistent edge cases.
Modern transcription-based answering machine detection lands in the 95 to 98 percent accuracy range with sub-three-second latency, compared to 70 to 85 percent for the legacy energy-and-silence heuristics that ship with most telephony providers. The accuracy ceiling depends on whether you tune for false positives (treating real humans as machines, which costs you the conversation) or false negatives (treating machines as humans, which leaves a confused message), and the right tradeoff varies by campaign type.
Three things running in parallel: a voice activity detector that watches for sound during agent playback, a streaming transcription that emits a partial transcript within about 100 milliseconds, and a semantic classifier that distinguishes a real interruption from a backchannel like "uh-huh." Without the third layer, the agent either ignores genuine interruptions or yields to every breath and acknowledgment, both of which feel wrong on the call.
Move the floor out of the system prompt and into a server-side function the language model cannot see. The agent can discuss pricing freely, but every committed quote runs through a function that validates against the floor before the number is spoken. This is the only pattern that survives adversarial users — prompt-based guardrails leak under sustained pressure, function-gated guardrails do not.
Under 700 milliseconds end-to-end, measured from the caller's last word to the agent's first word. Above 900 milliseconds, callers notice the gap and start to disengage. Retell AI publishes ~600ms latency as a measured production figure across its 40 million monthly calls, which sits comfortably below the conversational threshold and leaves headroom for function-call latency on top.
A defined escalation trigger fires and the call routes to a human with the full transcript, function-call history, and detected sentiment attached. Production deployments define these triggers explicitly — guardrail rejected twice, AMD verdict unsure twice, IVR navigation failed past a depth threshold, customer requested a human, sentiment dropped below a floor. The human picks up the call already knowing what was discussed, which is the difference between a warm handoff and starting over.
The function-gated guardrails and asynchronous AMD become essential at any volume where adversarial callers or voicemail rates are non-trivial — typically anything above a few hundred calls a day. Below that, the failure modes are real but the absolute count is small enough that human QA can catch them after the fact. The interruption-handling layer matters from call one regardless of volume, because every individual caller experiences it.
The four behaviors above are what production voice AI looks like underneath the conversation. Building them from scratch is roughly six to twelve months of engineering — WebRTC media handling, codec negotiation, SIP signaling for DTMF, a streaming transcription pipeline, a semantic interruption classifier, the function-call orchestration layer, and the post-call QA tooling that makes the whole thing improvable. Most teams that try end up shipping the easy parts and discovering the hard ones in production.
The shorter path is using infrastructure where these behaviors are already in place. The platforms now operating at 40 million-plus calls per month — Retell AI being the public reference point — exist because the cost of getting these four behaviors wrong, at scale, is what most call-center AI projects underestimate. If you want to hear how this architecture actually sounds on a real phone call, you can spin up a test agent at retellai.com with $10 in free credits and route a call through your own number to test the barge-in, IVR, and floor-price behaviors on something real rather than reading about them.
See how much your business could save by switching to AI-powered voice agents.
Total Human Agent Cost
AI Agent Cost
Estimated Savings
A Demo Phone Number From Retell Clinic Office
Start building smarter conversations today.
