Home

/

Glossary

/

What Is Voice AI?

What voice AI is, the speech recognition, language model and speech synthesis behind it, and how businesses use it to answer phone calls.

Voice AI is the group of artificial intelligence technologies that turn spoken language into meaning and meaning back into speech, so software can hold a real-time conversation with a person. It combines speech recognition, a language model and speech synthesis, plus the timing logic that makes a spoken exchange feel natural.

In business, voice AI often runs on phone calls. An AI voice agent is voice AI applied to a job: it answers or places a call, works out what the caller needs, looks things up in other systems, and either resolves the request or hands it to a person. To follow one call from ring to record, see how AI voice agents work on a call.

How does voice AI work?

A traditional voice AI system chains three models and repeats them on every turn:

  1. Speech-to-text (STT), also called automatic speech recognition, converts the caller's audio into text.
  2. A large language model (LLM) reads that text along with the conversation so far and decides what to say or do next.
  3. Text-to-speech (TTS) turns the reply into audio the caller hears.

This is a cascaded pipeline. A newer design, the speech-to-speech (S2S) model, takes audio in and produces audio out without writing text in between. Hybrid setups pair a voice model that runs the live conversation with a text LLM in the background that follows the instructions and calls tools.

The models are only part of the system. A phone conversation also needs software that detects when the caller has finished speaking (turn-taking and endpointing), lets the caller interrupt, filters background noise, and switches to a backup provider if one fails. Retell's orchestration overview notes that using the building blocks on their own "often results in high-latency, unnatural interactions".

Diagram of how voice AI handles one turn of a phone call. The caller's audio arrives over the phone network. In a cascaded pipeline, speech-to-text converts it to text, a large language model decides the reply and can call tools, and text-to-speech speaks the reply. A speech-to-speech model goes from audio in to audio out in one model. A hybrid pairs a live voice model with a background text LLM that runs instructions and tools. Underneath all three, an orchestration layer handles endpointing and turn-taking, interruptions, background noise filtering and provider fallback.
The three common voice AI designs and the orchestration layer around them, as described in Retell's documentation. Diagram by Retell AI, November 2026.

Voice AI technology: the building blocks

ComponentWhat it does on a call
Speech recognition (ASR or STT)Turns the caller's audio into text while they speak
Large language model (LLM)Interprets the request, writes the reply, and can call tools to look up or update records
Speech synthesis (TTS)Speaks the reply in a chosen voice
Speech-to-speech modelHears and speaks directly, with no text step in between
Turn-taking and interruption handlingDecides when to speak, when to stop, and when the caller is only pausing
Telephony and voice APIConnects the system to the phone network so it can place and receive calls

Voice AI vs IVR, voice assistants and voice APIs

  • IVR (interactive voice response) moves callers through fixed menus by keypad or set phrases. Voice AI handles requests in the caller's own words. See AI voice agents vs traditional IVR.
  • Voice assistants on phones and smart speakers use the same family of technology to carry out one person's commands.
  • A voice API places, receives and controls phone calls. The voice API carries the call; voice AI holds the conversation on it.

Where AI voice agents fit

Retell AI is an AI voice agent platform for contact centers. Its documentation says Retell runs the telephony, speech and language model layers, so a team configures the agent instead of connecting providers. Teams describe the task in a single prompt or a conversation flow, choose a transcription mode and a voice, and can run a cascading model, a speech-to-speech model or the hybrid GPT Live voice model. Developers who need their own model can connect a custom LLM over a WebSocket, while Retell still handles telephony, transcription, turn-taking and speech synthesis.

Frequently asked questions about voice AI

What is the difference between voice AI and an AI voice agent?

Voice AI is the technology: speech recognition, language models and speech synthesis. An AI voice agent is built on it to do a specific job on a call, such as booking an appointment, and can take actions in other systems while it talks.

Is voice AI the same as conversational AI?

Not quite. Conversational AI covers any back-and-forth with software, typed or spoken. Voice AI is the spoken part, which adds speech recognition, speech synthesis and the timing problems of live audio.

Why does latency matter in voice AI?

A caller hears every delay as silence. Each turn waits on transcription, the language model and speech synthesis, so delays add up. Retell's docs note that transcription latency alone depends on the provider, language, endpointing and the caller's speech. See where voice AI latency comes from.

Are AI voice calls regulated in the US?

Yes. In February 2024 the FCC ruled that calls made with AI-generated voices are "artificial" under the Telephone Consumer Protection Act, which holds them to the same standards as other artificial or prerecorded voice calls, including prior express written consent before telemarketing robocalls. Retell's TCPA guide covers outbound calls. Ask counsel how the rules apply to you; this entry is not legal advice.

Is voice AI free or paid?

It depends on the product. Retell AI's pay-as-you-go plan starts from $0.07 per minute and includes $10 in free credits, per its pricing page.

See voice AI on your own calls

Retell AI is a platform to build, test, deploy and monitor AI voice agents. Talk to the Retell team about a pilot.

Security & compliance for regulated deployments

How Retell handles audits, data retention and access control.

Add Retell AI as a preferred source on Google

Further Reading

Related terms

Explainer

AI-to-Human Handoff: How Voice Agent Call Transfers Work

Learn More
Explainer

Conversational AI Architecture for Voice Agents

Learn More

Table of contents

Revolutionize your call operation with Retell