Why Multilingual Voice AI Fails at the Accent Handoff

Code-switching is when a speaker moves between two languages inside one conversation, often inside one sentence, and it is the case a language-by-language test never covers. An agent can pass a Spanish test and an English test and still sound wrong the first time a caller says "Quiero cambiar mi appointment para el viernes." The giveaway is the accent handoff: the moment the agent hears or says an English word inside the other language and handles it the way no local speaker would.

This guide is for contact center teams, and the agencies and BPOs that build for them, putting voice agents in front of bilingual callers. It covers where code-switched calls break, the test to run before launch and the settings that fix what it finds.

What is code-switching, and why does it matter on the phone?

Code-switching is ordinary bilingual speech. A caller speaks mostly in one language and drops in words or phrases from another: a product name, "email", "link", "appointment", an account number read out in English. Those English words tend to carry the speaker's own accent, not an American one.

The callers who do this are not a niche. The U.S. Census Bureau counted 67.8 million people in the United States who spoke a language other than English at home in 2019, almost 1 in 5, and Spanish was 62% of them. Of those Spanish speakers, 39% said they spoke English less than "very well", which means most of them also speak English very well. Bilingual callers are the ones able to mix the two, and an agent built for them should expect it.

Where does a code-switched call break?

It breaks in four places, and each one only shows up where the two languages meet, so testing each language on its own misses all four.

• Hearing. The transcriber has to catch an English word said with a Spanish or Hindi accent in the middle of a non-English sentence. A single-language agent transcribes in one language only. A multilingual agent that crosses language families, such as English and Spanish, uses a speech pipeline that Retell's multilingual docs call "less accurate per language than single-language models."
• Reply language. One English word from the caller should not flip the agent into English. On a multilingual Retell agent, the docs say the agent "is allowed to respond in any of the selected languages and chooses based on what the caller speaks (and any instructions you give in the prompt)." The prompt is where you set the rule.
• Voice. This is the accent handoff. The same docs say the agent "detects the language of each response and uses the matching pronunciation", falls back to the first language you selected if detection fails, and that "not all voice providers handle accents." Because the docs describe language detection per response, the replies to listen to hardest are the mixed ones, where an English word sits inside a Spanish sentence.
• Handoff. When the call moves to another AI agent or to a person, the language has to move with it. On a Retell agent transfer, the destination agent's own language and voice apply unless you turn on "keep the current language". A caller who has been speaking Spanish and lands in an English-only queue starts over. See how AI-to-human call transfers work.

Closely related languages add one more risk. Retell's docs say speech recognition "struggles most on languages that share vocabulary, phonemes, or a writing system", with Cantonese and Mandarin as the example, and that "callers routinely get transcribed as the wrong Chinese variant, which then makes the agent respond in the wrong language."

How do you run a code-switching test before launch?

Build a small test set from how your callers actually talk, have native bilingual speakers say it, and score the switch points separately from everything else.

1. Collect the mixed phrases. From calls your bilingual staff already handle, list the English words callers drop into the other language: plan names, "email", "link", street names, account numbers, dates.
2. Use real accents. Have native speakers from your callers' regions place the test calls. A colleague with an American accent will not produce the accented English words your callers use, so the test passes for the wrong reason.
3. Script each pattern in the table below, in both directions.
4. Score the switched words on their own. An overall accuracy number can hide the problem. A paper accepted at ICASSP 2025 found that fine-tuning speech models on non-code-switched data improved standard metrics such as word error rate on code-switching test sets "although actual code-switched words worsen."
5. Have a native listener grade the voice. For each mixed reply: would a local bilingual speaker say it this way?
6. Agree the pass bar first. Decide which errors block launch, such as a wrong reply language or a misheard account number, and which go on the fix list.

Each Retell test method covers part of this. Simulation testing "runs as a text conversation", so it checks reply language and flow and can rerun as a batch after every change, but it cannot hear an accent. A web call test plays real audio ("pacing, pronunciation, and filler") and follows transfers to another agent, but transfers to a phone number fail there, so use a phone call test for those and for carrier audio. For the release process around these checks, see how to test and release voice agent changes.

How do you fix what the test finds?

Start with the narrowest language setup that serves your callers, then fix individual words. Retell's docs rank a single language most accurate, variants of one base language such as en-US and en-GB the same, and languages across families lower.

Two caveats. Pronunciation dictionaries depend on the voice model: Retell platform voices support IPA and CMU, Cartesia and Inworld voices IPA, while ElevenLabs eleven_multilingual_v2, eleven_flash_v2_5 and eleven_v3 do not. An ElevenLabs voice set to Auto on a non-English agent resolves to a model without dictionary support, so a bilingual agent on that setting cannot use one. The docs also warn that overriding common words "can produce unnatural speech." And speech recognition providers differ: Retell's provider page lists Deepgram (for supported language combinations), Soniox and AssemblyAI with code-switching support and Azure as single-language only.

What should you watch after launch?

Live calls will show switch patterns your test set missed. Add a post call extraction field, such as a selector for the language the caller used, to pull out the bilingual calls, then run AI QA on a sample and read each transcript at the switch points. The voice agent failure modes guide helps separate a transcription error from a wrong answer, and the QA scorecard template gives the review a fixed shape.

Every mixed phrase that trips the agent goes into the test set before the next change ships. If two voices both pass with native listeners, A/B testing can split live traffic between two agents by percentage.

Frequently asked questions

What is code-switching in voice AI?

Code-switching is when a caller moves between two languages in one call or one sentence, such as a Spanish sentence with an English word in it. The agent has to transcribe both, answer in the right one and pronounce the mixed words like a local speaker.

Should one agent handle both languages, or should I use one agent per language?

Use one agent per language, or set the language per call, whenever the dialed number, CRM data or a language-selection step tells you the caller's language; Retell's docs rank a single language most accurate. Use a multilingual agent when you cannot know the language ahead of time, with only the languages your callers use.

Can automated simulation tests catch code-switching problems?

Part of it. Retell's simulation tests run as text, so they check which language the agent answers in, but not accent or pronunciation. Use web calls with native bilingual speakers for the voice and phone calls for transfers to a person.

Why does a multilingual agent sound fine in each language but wrong in mixed speech?

The switch points fail while each language works on its own. The transcriber can miss an accented English word, the agent can reply in the wrong language, and the docs describe the voice detecting language per response, so mixed replies are the ones to check. Score switched words and mixed replies separately.

How do I fix a word the agent keeps mispronouncing?

Add a pronunciation dictionary entry in IPA or CMU and use Test Pronunciation to hear it in the agent's voice. Check first that your voice model supports dictionaries, since some multilingual models do not.

Test bilingual calls with Retell AI

Retell supports single-language agents, multilingual agents with an exact set of languages, and one agent per language joined by agent transfer, testable with simulation (except custom LLM agents), web calls and phone calls before callers hear it. See Retell's multilingual AI voice agents and the bilingual virtual receptionist, compare vendors in the 8 best multilingual AI voice agents, or start with multilingual setup basics.