Speculative decoding is a technique for speeding up a large language model: a small draft model guesses the next few tokens, and the large model checks them all in a single pass and keeps the ones it would have produced itself.
The method comes from two research papers. Google Research named it in Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman and Matias, first posted November 2022). DeepMind described its own version, developed separately, in Accelerating Large Language Model Decoding with Speculative Sampling (Chen and colleagues, February 2023).
Why is standard decoding slow?
Standard decoding is slow because a large language model writes one token at a time, and each token needs a full run of the model. The Google paper states it in one line: "decoding K tokens takes K serial runs of the model".
Many of those tokens are easy to predict. The same paper observes that "hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models".
How do the draft model and the full model work together?
The two models work in rounds, and each round has three parts:
- Draft. The small model writes a short run of guesses, one token after another. The Google paper calls it the approximation model; DeepMind calls it the draft model.
- Verify. The large model, called the target model, scores every guessed position in parallel, in one pass.
- Accept or correct. Guesses are accepted from left to right. At the first rejected guess, the large model supplies its own token and the later guesses are thrown away. If every guess is accepted, the large model adds one more token.
Each pass of the large model yields at least one token and at most the number of guesses plus one, so it never needs more passes than standard decoding.
Worked example (illustrative). A 12-token reply, counting each word and the period as one token, with three guesses per round:
| Pass of the large model | Draft model guesses | Large model's verdict | Tokens kept |
|---|---|---|---|
| 1 | Your appointment is | All three accepted, adds "confirmed" | 4 |
| 2 | for Monday at | "for" accepted. "Monday" rejected and replaced with "Friday". "at" discarded | 2 |
| 3 | at ten in | All three accepted, adds "the" | 4 |
| 4 | evening . | "evening" rejected and replaced with "morning". The period is discarded | 1 |
| 5 | . | Accepted, and the reply ends | 1 |
| Total | 12 tokens in 5 passes |
Standard decoding needs 12 passes of the large model for this sentence. Here it runs 5 times, an average of 2.4 tokens per pass.
Why is the output unchanged?
The output is unchanged because the large model has the final say on every token. With greedy decoding, where the model always takes its top choice, a guess is kept only if it is the token the large model would have picked. The vLLM documentation lists a test for exactly this: greedy sampling with speculative decoding "matches greedy sampling without it".
With random sampling, a guess is accepted with a probability set by how the two models rate it, and a rejected guess is redrawn from an adjusted distribution. The Google paper proves this leaves the output distribution unchanged. DeepMind's wording is that the scheme "preserves the distribution of the target model within hardware numerics".
Some newer variants relax the check on purpose to gain speed. vLLM notes that one of its verification methods "is not distribution preserving".
When does speculative decoding help, and when does it not?
Speculative decoding helps when the draft model is cheap and usually right, and the server has spare compute.
| Condition | Effect on speed | Source |
|---|---|---|
| Guesses are usually accepted | More tokens per pass. Expected tokens per pass = (1 − α^(γ+1)) ÷ (1 − α), where α is the acceptance rate and γ the guesses per round | Google paper |
| Draft model is far smaller than the target | Low drafting cost. A draft around two orders of magnitude smaller "usually performed best" | Google paper |
| Spare compute is available | Required. The method assumes "enough compute resources to support the increased concurrency" | Google paper |
| Heavy traffic on the server | Smaller gain. vLLM rates a draft model "High gain" at low request rates and "Medium gain" at high ones | vLLM documentation |
| Draft is wrong too often for what it costs | No gain. A speedup exists only when the acceptance rate is higher than the draft's cost relative to the target | Google paper |
As an illustration, an acceptance rate of 0.6 with three guesses per round gives about 2.2 tokens per pass.
Published speedups: the Google paper reports 2X to 3X on an 11-billion-parameter T5-XXL model. DeepMind reports 2 to 2.5 times on the 70-billion-parameter Chinchilla with a 4-billion-parameter draft. Results depend on the models, the task and the hardware.
A production example from voice: Tell-1, Retell's own LLM, uses speculative decoding to get about four tokens out of each pass of the full model, according to Retell's technical post on Retell LLM.
Why does speculative decoding matter for real-time voice?
Speculative decoding matters for real-time voice because a caller hears model time as silence. On a call, the model is one stage of the speech-to-text, LLM and text-to-speech pipeline, and its delay adds to the call's latency.
Retell's custom LLM best practices explain which part of model time counts: Retell "starts speaking as soon as it has your first complete sentence", so the number to watch is "time to first token plus the time to finish that first sentence". Speculative decoding works on the second part: once generation starts, tokens arrive in groups. It does not shorten the prompt read that comes before the first token. For that, Retell's latency troubleshooting guide starts with model choice and prompt length.
The other reason is accuracy. Moving to a smaller model also cuts delay, but the same guide warns that switching models "changes your agent's behavior, not just its speed". Exact speculative decoding keeps the large model's output.
Whoever serves the model applies it, so an agent builder sees it as model speed. On Retell, Tell-1 is served this way. To compare models on speed and call quality, see the best LLMs for voice AI agents.
Frequently asked questions about speculative decoding
Is speculative decoding lossless?
In its exact form, yes. The original papers prove the output distribution matches the large model's, and vLLM describes it as "theoretically lossless up to the precision limits of hardware numerics". Variants that relax verification are not lossless.
Who created speculative decoding?
Yaniv Leviathan, Matan Kalman and Yossi Matias of Google Research posted the paper in November 2022 and presented it at ICML 2023. A DeepMind team published its own version, called speculative sampling, in February 2023.
What is a draft model?
A draft model is the small, fast model that writes the guesses. Hugging Face's documentation says the method works best when it "is significantly smaller than the main model and uses the same tokenizer".
Does speculative decoding reduce cost?
It can go either way. The Google paper notes that the total number of arithmetic operations "might be higher than that of the standard decoding algorithm", since rejected guesses are wasted work. Google's 2024 retrospective adds that faster results on the same hardware mean "fewer machines are needed for serving the same amount of traffic".
Is multi-token prediction the same as speculative decoding?
No. Multi-token prediction (MTP) is one way to produce the guesses. vLLM's documentation lists MTP next to draft models as a speculation method.
Test model speed on your own calls
Retell AI is an AI voice agent platform for contact centers. Talk to the Retell team about a pilot on real calls.
Security & compliance for regulated deployments
How Retell handles audits, data retention and access control.
Further Reading
Related terms
Table of contents
