Retell LLM: Interleaved Reasoning for Real-Time Voice Agents

The problem

Phone calls need a different kind of model

A great phone agent has to reason well, sound like a person and answer quickly, all on the same turn.

Retell agents book appointments, run intake, make sales calls, answer front desks, handle support and collect payments. Each agent follows a script its builder wrote, with long, detailed steps, rules and tools. On every turn the model has to know where it is in that script, make sense of a messy transcript, and decide whether to ask, answer or act.

1OpenAI. Learning to reason with LLMs. 2024.2Snell et al. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. 2024.3DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. 2025.Most models get smarter by thinking longer.^1,2,3 That may work well enough in a chat window, but on a phone call, every second of thinking is a second of silence.

How it works

It thinks while it talks

We designed a new model that interleaves speaking and reasoning on every turn.

The model writes the first sentence of its reply right away. While that sentence is playing, it reasons silently: which rule in the script applies, what is still missing, and whether it is time to call a tool. Then it continues the reply, grounded in that reasoning. The caller hears one unbroken answer, and the reasoning is never spoken.

It is built for the contact center, where every turn runs on a long script, real tools and strict rules.

Training

Rewarded for what callers notice

Retell LLM is trained in three stages on curated production conversations. The last, reinforcement learning, rewards the four things a call-center model must get right: correct tool calls, no hallucination, instruction following and sounding natural.

Four rewards for a good call turn

For each moment in a call, the model writes eight responses and every one is scored. Code and judges decide whether a turn is right; two auxiliary rewards shape how it sounds and how long it thinks.

Code harness. Calls to tools outside the agent's toolbox, malformed output and other critical errors are caught in code before any judge sees it.

Correctness. Multiple judge models grade tool-call correctness, hallucination and instruction following, and their verdicts are pooled into a single reward signal.

Naturalness. Naturalness is easier to compare than to score. A judge ranks all eight rollouts side by side against a rubric, which provides more useful insight than grading each alone.

Reasoning length. Shortens reasoning without losing intelligence. Long reasoning is penalized only on prompts the model already solves reliably, meaning it stays brief where it can and keeps thinking where it must.

Keeping the lead alive

Midway through RL, the model drifted back to think-first: an empty lead, then reasoning, then the reply. The caller was back to silence. We call this lead collapse. The choice to speak or stay silent is made at the very first token, which makes it effectively a classification at that position, so we added a lead bonus on that token alone. The bonus carries an inherent trade-off between latency and accuracy: speaking sooner means committing to the lead before any reasoning has happened.

Credit assignment

Each reward trains only the tokens it depends on. Naturalness skips tool-call tokens, so a well-formed tool call is never judged on how it sounds, but it still trains the reasoning, since what the model thinks shapes how the reply sounds. The lead bonus is applied only to the first token.

Finding the learning signal

Before RL, we measured pass@k: how often at least one of k sampled answers passes. On 200 conversations from our instruction-following eval, the model passed 77.5% of the time with one try, 89.5% with four and 93.0% with eight. The right answer was often already within reach; it just wasn't the model's first answer. Moving it there is RL's job.

4Yu et al. DAPO: An open-source LLM reinforcement learning system at scale. 2025. See dynamic sampling.Group-relative RL only learns from disagreement: when all eight rollouts pass, or all fail, every advantage is zero and the prompt contributes no gradient.⁴ We therefore pre-filter the training set to keep the difficult cases, where the model is still inconsistent, so RL spends its compute on turns where it can actually improve.

Inference optimization

Fast, stall-free streaming

Retell LLM reasons while its first sentence is still playing, which means the rest of the reply has to be ready before that sentence ends. Our inference stack is built for exactly that: fast tokens and no stalls.

No stalls. Under realistic load, about one reply in three used to stall, with more than 100 ms between two tokens: whenever a GPU read in a new prompt, however short, every reply streaming on it froze. Running every step on CUDA graphs, and reading long prompts in as small slices, reduced stalled replies from 34% to 0.3%.

Call-aware routing. Every turn resends the whole conversation, so we keep each call on the GPUs that already hold it in cache, using session affinity. That nearly doubles the share of prompt tokens served from cache, from 36% to 69%, ensuring that each turn has less to read in and replies start sooner.

Fast decoding. Speculative decoding gets about four tokens out of each pass of the full model, and fused, overlapped kernels trim the fixed cost of every step. Under heavy load, Retell LLM takes a median of 3.2 ms per token, more than 300 tokens a second for every reply.

What callers hear

When people talk, a pause of around 400 ms between sentences sounds natural. Retell LLM's pause between its first sentence and the rest of the reply almost always stays below that, so callers hear a natural reply instead of a model stopping to think.

How we measured: one test server replaying realistic multi-turn call traffic under sustained heavy load. Times are model time only and exclude network, speech recognition and speech synthesis. Before-and-after comparisons replay the same traffic.

Benchmarks

Built for calls, measured on public benchmarks

We test Retell LLM on public agent, instruction-following and reasoning benchmarks it never trained on.

We also rank it on the Retell vCX benchmark, which scores tool calling, hallucination, instruction following and natural phone speech on contexts drawn from real calls.