Session affinity, also called sticky sessions or session persistence, is a load balancing technique that sends every request from the same client or session to the same backend server, so state held in that server's memory stays reachable.
Without it, each request is placed on its own. AWS says an Application Load Balancer by default "routes each request independently to a registered target", and that the sticky session feature lets it "bind a user's session to a specific target". That matters when a server keeps something locally that the next request needs: a login session, a shopping cart or a cache.
How do load balancers implement session affinity?
A load balancer needs a way to recognize the session and a rule that maps it to a server. Three methods cover most products.
Cookie. The load balancer sets a cookie on the first response and reads it on later requests. AWS's duration-based stickiness uses a cookie named AWSALB with a lifetime you set between 1 second and 7 days. HAProxy uses a cookie to "remember which server the client was routed to before". The client has to support cookies.
IP hash. The client's IP address picks the server. NGINX's ip_hash uses "the first three octets of the client IPv4 address, or the entire IPv6 address" as the hashing key. A Kubernetes Service offers the same idea as sessionAffinity: ClientIP, with a default timeout of 3 hours. Neither needs a cookie, but clients behind one shared address land on the same server, and with NGINX so does everyone who shares those three octets.
Consistent hashing. The load balancer hashes a property of the request, such as a header or a session ID, onto a ring of servers. Envoy says that with a well-partitioned ring, adding or removing one host from a set of N "will affect only 1/N requests". NGINX's hash directive has a consistent option that, per its docs, "helps to achieve a higher cache hit ratio for caching servers".
| Method | What identifies the session | Documented example | Main limit |
|---|---|---|---|
| Cookie | A cookie set by the load balancer or the application | AWS Application Load Balancer, HAProxy | The client must support cookies |
| IP hash | The client's IP address | NGINX ip_hash, Kubernetes ClientIP | Clients behind a shared address pile onto one server |
| Consistent hashing | Any request property you choose | Envoy ring hash and Maglev, NGINX hash ... consistent | Every request must carry a stable key |
| Prefix-aware routing | The opening tokens of an LLM prompt | OpenAI prompt caching, Ray Serve | Load still counts, so a busy server can lose the request |
What are the trade-offs of sticky sessions?
Session affinity gives up some balance and resilience in exchange for keeping state close.
- Load can skew. Google Cloud says affinity "works well when there is a reasonably large distribution of unique connections". With few or uneven sessions, some servers carry more, which works against scalability.
- A failed server ends the affinity. On AWS, if a target fails the load balancer "chooses a new healthy target" and keeps the session there "even if the failed target comes back". Anything held only in the old server's memory has to be rebuilt.
- It is best effort. Google Cloud provides session affinity "on a best-effort basis" and lists health check changes, added or removed backends and backend fullness as events that can break it.
HAProxy's docs give the standard advice: "Often, it is optimal to avoid session persistence altogether by building your services to be stateless", for example by storing state in a shared database. Affinity is for the state you cannot move cheaply. A cache is the usual case.
Why does session affinity matter for LLM serving?
It matters because the cache that makes a repeated prompt cheap lives on one machine. A model server keeps the computed state of a prompt, the KV cache, in memory. vLLM reuses it when a new query "shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part".
That only works if the next request reaches the same machine. OpenAI's prompt caching guide states it directly: "Cached states live on individual machines", and "a request can reuse a cached prefix only if it reaches a machine holding a matching entry that has not expired". OpenAI routes on machine load, a hash of the prompt's initial tokens and, on older models, a cache key you supply. For self-hosted models, Ray Serve ships a router that "routes requests with similar prefixes to the same replicas, maximizing KV cache hit rates", and falls back to plain load balancing when queues are uneven.
Phone calls fit this pattern closely. On every turn of a multi-turn conversation, a voice agent sends the large language model its instructions plus the transcript so far, because the model does not keep the call between requests. Each request repeats most of the one before it. So the call is the session and the call ID is a natural key. A turn that lands on a different server processes the whole prompt again, which adds to latency.
Tell-1, Retell's own LLM, is served this way. The Retell LLM technical post calls it call-aware routing: every turn resends the whole conversation, so each call is kept on the GPUs that already hold it in cache. Per the post, that raised the share of prompt tokens served from cache from 36% to 69%.
Where session affinity shows up on Retell
It shows up when you run your own model server. With a custom LLM, Retell "opens a WebSocket to your server for each call", with the call ID as the last path segment of the URL. AWS notes that "WebSocket connections are inherently sticky", so a call stays on one of your servers while its socket is open.
Retell's best practices recommend keeping "explicit per-call state on your server, keyed on the call ID from the WebSocket URL". Retell also rebuilds a connection that drops mid-call, and if you run several servers behind a load balancer, the new socket can reach a different one. To keep the state reachable, route on the call ID in the path, or keep that state in a shared store. IP hash will not spread the load here: the setup guide lists a single outbound IP address for this connection, so every call hashes to the same server.
Frequently asked questions about session affinity
Is session affinity the same as sticky sessions?
Yes. AWS describes its sticky session feature as "also known as session affinity". Session persistence is a third name for the same thing.
What happens to a sticky session when the server fails?
The load balancer picks another healthy server and the session sticks there. State that lived only in the failed server's memory is lost unless it was also stored somewhere shared.
Do WebSocket connections need sticky sessions?
Not while the connection is open, because every message travels over the same connection to the same server. Affinity matters again when the client reconnects.
Why does session affinity matter for prompt caching?
A prompt cache sits in the memory of the machine that built it. A follow-up request gets the cached prefix only if it is routed to that machine.
Bring your own model to live calls
Retell AI is an AI voice agent platform for contact centers. Retell handles telephony, transcription, turn-taking and speech, and the custom LLM integration connects your model server over one WebSocket per call.
Security & compliance for regulated deployments
How Retell handles audits, data retention and access control.
Further Reading
Related terms
Table of contents
