Prompt caching is a technique that lets a large language model skip work it has already done: when a new request starts with the same tokens as an earlier one, the model reuses the saved computation for that shared start and only processes what is new. The result is a faster first token and, on most APIs, a lower price for the reused tokens.
The reused part is called the prefix. OpenAI's guide defines it as "the unchanged tokens at the beginning of a prompt". Instructions, tool definitions and earlier conversation turns usually sit there.
How does prompt caching work?
Anthropic's documentation describes the sequence: the system checks whether a prompt prefix is already cached from a recent query, uses the cached version if it is, and otherwise processes the full prompt and caches the prefix.
Three rules follow from that design:
- The match is exact. Anthropic states that cache hits require "100 percent identical prompt segments". One changed character early in the prompt means everything after it is processed again.
- Order matters. Stable content goes first and changing content goes last.
- Entries expire. Anthropic's default cache lifetime is 5 minutes, refreshed each time the cached content is used. OpenAI's lifetimes depend on the model.
How do prompt caching, prefix caching and the KV cache relate?
They are three layers of the same idea. The KV cache is the data, prefix caching is the reuse of that data across requests, and prompt caching is the name API providers give the feature.
| Term | What it is | Scope | Where you see the term |
|---|---|---|---|
| KV cache | The key and value states a model computes for each token it has read, kept so they are not recomputed for every new token | Inside one request | Model internals, inference engines |
| Prefix caching | Keeping the KV cache after a request ends and reusing it for a later request that starts with the same tokens | Across requests on the same server | Inference engines such as vLLM |
| Prompt caching | Prefix caching offered as an API feature, with rules for lifetime, minimum length and price | Across API calls | Model provider documentation and pricing pages |
A Hugging Face explainer puts the KV cache simply: "Instead of recomputing everything from scratch, the model reuses what it has already calculated". The vLLM documentation says its automatic prefix caching "caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix". OpenAI confirms the link from the API side: "The prompt cache stores key-value (KV) tensors, not the tokens themselves."
Why does prompt caching matter on a multi-turn phone call?
A voice agent calls the model once per turn, and each request carries the instructions, the tool definitions and the conversation so far. In a multi-turn conversation nearly all of that is identical to the previous turn's request. Only the last exchange is new. Without caching, the model reads the same instructions again on every turn.
Worked example (illustrative numbers). An agent has 2,500 tokens of instructions and tools, each turn adds 100 tokens of conversation, and every turn after the first finds its prefix in the cache:
| Turn | Prompt tokens | Reused from cache | New tokens to read |
|---|---|---|---|
| 1 | 2,600 | 0 | 2,600 |
| 2 | 2,700 | 2,600 | 100 |
| 5 | 3,000 | 2,900 | 100 |
| 10 | 3,500 | 3,400 | 100 |
| All 10 turns | 30,500 | 27,000 | 3,500 |
Across the call, 27,000 of 30,500 prompt tokens (about 89%) come from the cache. From the second turn on, the model reads 100 new tokens instead of more than 2,600.
That matters for two reasons. Time to first token is part of the latency a caller hears as silence, and reading a long prompt takes time. Cost is the second: Anthropic prices cache reads at 0.1 times the base input token price, with cache writes at 1.25 times for the 5-minute lifetime.
What stops a cache hit?
- A changed prefix. A timestamp, a caller's name or a reordered tool list near the top of the prompt changes every token after it.
- A gap longer than the cache lifetime.
- The request reaching a different machine. OpenAI's guide notes that "cached states live on individual machines" and that a request can reuse a prefix "only if it reaches a machine holding a matching entry that has not expired".
Routing can close that last gap. Tell-1, Retell's own LLM, keeps each call on the GPUs that already hold it in cache, using session affinity. According to the Retell LLM technical post, that nearly doubles the share of prompt tokens served from cache, from 36% to 69%, so each turn has less to read in and replies start sooner.
Caching does not change the answer. The vLLM design notes say prefix caching "won't change model outputs".
Where AI voice agents fit
Retell's latency troubleshooting guide states the cost of prompt length directly: "Every token in your prompt adds to the time-to-first-token, and that cost is paid on every turn of the conversation."
If you connect your own model through Retell's LLM WebSocket, each response request arrives with the "complete live transcript collected in the call so far". Your server builds the prompt, so cache behavior depends on how you build it and on your model provider: keep instructions and tools first and unchanged, and append the transcript after them. For the full path a turn takes, see how real-time voice AI works.
Frequently asked questions about prompt caching
What is the difference between prompt caching and the KV cache?
The KV cache is the stored attention state a model builds while reading tokens. Prompt caching keeps that state after a request ends so a later request with the same opening tokens can reuse it.
Is prefix caching the same as prompt caching?
They are the same mechanism. Prefix caching is the term inference engines use. Prompt caching is the product name model providers use for it in their APIs.
Does prompt caching change the model's response?
No. It only skips recomputing the shared prefix.
How long does a prompt cache last?
It depends on the provider. Anthropic's default is 5 minutes, refreshed on each use, with a 1-hour option at a higher write price. OpenAI's lifetimes depend on the model.
Why does prompt caching matter for voice agents?
Every turn of a call resends the instructions and the conversation so far. With a cache hit, the model reads only the newest exchange.
Hear the difference on your own calls
Retell AI is an AI voice agent platform for contact centers. Tell-1 is its own LLM, built for phone calls. Its docs show how to check the latency of every call by component, so you can see how much of the wait is the model. Talk to the Retell team.
Security & compliance for regulated deployments
How Retell handles audits, data retention and access control.
Further Reading
Related terms
Table of contents
