AI Agent Benchmarks Explained for Voice Agents

An AI agent benchmark is a fixed set of tasks with a scoring rule, used to compare models on the same work. For AI voice agents, four are worth knowing: τ²-bench (tau2-bench) for tool use under a policy, Multi-IF for following instructions across turns, GPQA Diamond for hard reasoning, and the Retell vCX Benchmark for contact center calls. Each one measures a single slice of what happens on a phone call.

This guide explains what each benchmark tests, how it is scored, and what the score does not tell you. Retell AI publishes this guide and also publishes one of the four benchmarks, the vCX Benchmark. Scores change with every model release, so each section links to the source that carries current results.

What is an AI agent benchmark?

An AI agent benchmark gives every model the same tasks and the same grader, then reports how often the model gets the task right. What a score means depends on the task, on who or what plays the other side of the conversation, and on how a pass is decided.

Graders come in three kinds. A state check compares the result of the agent's actions with a known correct result, such as a database after a refund. A rule check runs a script against the reply. A judge model grades the reply against a rubric. State and rule checks are repeatable but narrow. Judge models can grade tone and meaning, and they carry their own blind spots.

What do the four benchmarks measure, side by side?

They cover different skills, and only the vCX Benchmark is built from phone calls. The table summarizes each one from its paper, repository or leaderboard, read on November 13, 2026.

What does τ²-bench measure in its airline and retail domains?

τ²-bench measures whether an agent can finish a customer service task through conversation while following a written policy and calling the right tools. A language model plays the customer, and the agent works against a mock database of orders or reservations. The original τ-bench paper built the two domains with 115 retail tasks and 50 airline tasks. The τ²-bench paper reuses verified versions of both and adds a telecom domain.

Scoring is strict. According to the evaluation notes in the repository, each task's reward is the product of its checks. By default those are a database check, which compares the final database with the target state, and a communication check, which confirms the agent gave the customer the required information. Any sequence of tool calls that reaches the right end state passes.

Two reported numbers matter. Pass^1 is the share of tasks solved in a single attempt. Pass^k is the share solved in all of k repeated attempts, a metric the original paper introduced to capture consistency. That 2024 paper reported that even leading function-calling agents of the time succeeded on fewer than half the tasks, with pass^8 under 25% in retail.

What it does not tell a voice buyer: the airline and retail text scores come from typed conversations with no audio, no interruptions and no clock. The policies and tools are the benchmark's own. The simulated customer is also a model, and the τ²-bench authors measured simulator error rates of 40% in retail and 47% in airline, with 12% and 13% serious enough to block the task. The same repository now ships separate voice benchmarks, and its 2.0 release adds six languages and says its scores are not comparable with the 1.x leaderboard. Check which version a quoted score comes from.

What does Multi-IF measure?

Multi-IF measures whether a model keeps following instructions as a conversation goes on, and whether that holds outside English. The Multi-IF paper from Meta extends an earlier single-turn test, IFEval, into 4,501 conversations of three turns each across eight languages: English, French, Russian, Hindi, Italian, Portuguese, Spanish and Chinese.

The instructions are written so they can be verified, such as a ban on commas or a rule to answer in capital letters. The headline score averages four accuracy figures. Instruction-level accuracy counts each instruction on its own. Conversation-level accuracy counts a conversation only if every instruction up to that turn was followed. Each is reported under a strict and a loose criterion.

The paper's main finding is the reason this benchmark matters for phone agents: every one of the 14 models tested failed more often with each added turn. One model dropped from 0.877 on the first turn to 0.707 on the third. An agent that follows a rule at the start of a call and drops it a few exchanges later fails in the same way.

What it does not tell a voice buyer: the instructions are formatting and writing constraints on text, such as length, punctuation and keywords. They are not business rules. There are no tools, three turns is short next to a real call, and nothing is spoken aloud.

What does GPQA Diamond measure?

GPQA Diamond measures whether a model can answer very hard science questions that a web search does not solve. The GPQA paper describes 448 multiple-choice questions in biology, physics and chemistry, written and validated by people who have or are pursuing PhDs. Diamond is the 198-question subset where both expert validators answered correctly and most non-experts did not.

Scoring is plain accuracy. Each question has four options, so random guessing scores 25%. On the full set, experts reached 65% and skilled non-experts with unrestricted web access reached 34%.

What it does not tell a voice buyer: almost everything about a call. There is no conversation, no instruction to follow, no tool and no time limit. It is a proxy for reasoning depth. With 198 questions, small gaps between models are also weak evidence, and the authors list the dataset's small size as a limitation.

What does the Retell vCX Benchmark measure?

The Retell vCX Benchmark measures how models handle contact center call turns across four suites. The live leaderboard, read on November 13, 2026, describes them this way:

• Instruction following: whether the model honours role constraints, business rules and refusal rules.
• Grounding: whether answers stay within the prompt, tools and retrieved context.
• Tool calling: whether the model selects the right action and passes valid arguments.
• Colloquial: whether replies sound natural, concise and safe to speak aloud.

Each suite reports a pass rate over roughly 1,000 or more cases, and the overall score is the average of the four. The leaderboard listed 62 entries across 45 models on that date and shows median latency and cost per minute beside each score.

The published method note is the vCX-Hard post, which covers the first two test sets, grounding and tool calling. Cases were mined from production calls, sampled across organizations so that no single customer dominates. A panel of judge models scores each case against a rubric and a reference solution. That post treats the ranking, more than the exact number, as the signal.

What it does not tell a voice buyer: it is a vendor benchmark, and the vendor has an entry. Tell-1, Retell's own model, ranks #1 overall on the Retell vCX Benchmark (as of November 2026). The cases come from Retell's traffic, so your call types may be thin or absent. The four suites score replies and tool calls, so speech recognition errors, interruptions and whether a whole call ends resolved are outside them. Judge models can also be wrong.

What do benchmark scores leave out for an AI voice agent buyer?

Benchmark scores leave out the platform around the model and the conditions of your own calls. A phone call adds speech recognition, turn-taking, speech synthesis, telephony and latency at every stage. None of the four benchmarks, in the form described here, scores those.

Results can differ by platform even when the agent is identical. In a study by testing vendor Cekura, one agent was deployed unchanged on six voice platforms and workflow scores ranged from 65.2% to 95.7%. Retell's summary of that Medicare study covers the method and the disclosure that applies to it.

How should a buyer use benchmark scores?

Use benchmark scores to build a shortlist, then decide on your own calls. A sensible order:

1. Match the benchmark to the risk: tool use results if a wrong action is the costly failure, instruction following if the agent must hold a long script.
2. Check the version, the date and who ran the test.
3. Prefer repeated-trial numbers such as pass^k when they are reported.
4. Rebuild the test on your own scenarios, and count a scenario as passed only when repeated runs all pass.

On Retell, simulation testing lets an AI play the caller while success criteria grade each run pass or fail, and batch tests rerun saved cases after every prompt change. After launch, AI QA scores a sample of real calls on measures such as hallucination and tool usage, and A/B testing splits live traffic between two agents. For model choice, see which LLM should power your voice AI agent. For the buying process, see how to evaluate an AI voice agent platform.

A benchmark tells you which models deserve a trial. A pilot on real calls tells you which one to run. Retell is an AI voice agent platform for contact centers, priced from $0.07 per minute, so you can prove it on your own calls before you sign anything.

Frequently asked questions

What is the difference between pass^1 and pass^k?

Pass^1 is the share of tasks an agent solves in a single attempt. Pass^k is the share it solves in every one of k repeated attempts on the same task. Pass^k is always equal to or lower than pass^1, and the gap shows how consistent the agent is.

Is τ-bench the same as τ²-bench?

No. τ-bench is the original 2024 benchmark with airline and retail domains. τ²-bench is the 2025 follow-up from the same group, which keeps verified versions of those two domains and adds a telecom domain where the simulated customer also uses tools.

Why is a science test like GPQA Diamond used to compare models for customer conversations?

GPQA Diamond is used as a general measure of reasoning ability. It says nothing direct about conversations, tools or speech. Read it as a check on reasoning depth, and rely on task benchmarks for the rest.

Which benchmark matters most for an AI voice agent?

None of them alone. Tool use under a policy, instruction following across turns and grounding map most closely to contact center work, so τ²-bench, Multi-IF and the vCX Benchmark are the closest fits. A graded test on your own call scenarios matters more than any of them.