LLM as a judge is an evaluation method in which one large language model scores or ranks the output of another AI system against written criteria, in place of a human reviewer. Teams use it to grade open-ended work, such as a phone conversation, that has no single correct answer.
The name comes from a 2023 paper, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Its authors reported that strong judges such as GPT-4 matched human preferences with "over 80% agreement, the same level of agreement between humans". They also listed its limits: position, verbosity and self-enhancement biases.
How does an LLM judge score a conversation?
An LLM judge reads the conversation and a set of instructions, then returns a verdict in one of three forms.
| Method | What the judge does | What comes back | Where it fits |
|---|---|---|---|
| Rubric grading | Reads one transcript against written criteria | Pass or fail, or a score, with a reason | Production QA and regression tests |
| Pairwise ranking | Compares two or more replies to the same turn | A winner or a ranked order | Choosing between prompts or models |
| Judge panel | Several judge models grade the same item and the verdicts are pooled | One pooled score | Scores that one model's bias could skew |
Rubric grading. The G-Eval paper had the judge reason through the criteria and then fill in a scoring form. On summarization, its scores correlated with human ratings better than earlier methods.
Pairwise ranking. Asking which of two replies is better suits qualities like tone, where a difference is easier to see than to score.
Judge panels. In Replacing Judges with Juries, a panel of smaller models from different model families "outperforms a single large judge, exhibits less intra-model bias" and was "over seven times less expensive".
How is LLM as a judge used in voice agent QA and benchmarks?
Voice agent QA uses an LLM judge for the parts of a call that only reading the transcript can settle: whether the agent followed its rules, stayed within its sources, and resolved the request.
Retell's docs show both common setups. In AI QA, an AI-evaluated condition is "a qualitative check the AI runs against each call's transcript and context", written as "the prompt the AI uses to judge whether the condition is met". In simulation testing, an AI plays the caller and "each run is graded pass or fail with a written explanation".
Benchmarks report the same kind of verdict at scale. The Retell vCX Benchmark publishes each suite score as a pass rate, "the percentage of tested scenarios it handled correctly", across instruction following, grounding, tool calling and colloquial speech (read November 13, 2026).
To turn verdicts into a weekly review, see the voice agent QA scorecard.
What are the known failure modes of an LLM judge?
An LLM judge fails in four documented ways. Each has a known check.
- Position bias. In Large Language Models are not Fair Evaluators, swapping the order of two answers let one model win "66 over 80 tested queries". Run each comparison in both orders.
- Self-preference. A 2024 study found an evaluator "scores its own outputs higher than others' while human annotators consider them of equal quality". Use a judge from a different model family, or a panel.
- Weak agreement with humans. Judging the Judges tested thirteen judge models and found "only the best (and largest) models achieve reasonable alignment with humans", with "a tendency toward leniency". Grade a sample by hand and compare.
- Drift. The model behind an API can change: one study measured GPT-4 falling from 84% to 51% accuracy on the same task in three months. Criteria move too. Who Validates the Validators? calls this criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria". Pin the judge version and re-run a fixed, hand-labeled set after any change.
Judges are also not deterministic. Retell's simulation docs advise: "Judge a scenario on its pass rate across runs, not one run."
When do code checks beat an LLM judge?
Code checks beat an LLM judge whenever the answer can be verified instead of read. Anthropic's guide to agent evals recommends "choosing deterministic graders where possible, LLM graders where necessary". It lists code-based graders as fast, cheap, objective and reproducible, and model-based graders as non-deterministic.
On a call, code can check that the tool the agent called exists, that its arguments are valid, and that response time stayed under a threshold. tau-bench grades whole conversations this way, comparing "the database state at the end of a conversation with the annotated goal state".
Retell's AI QA keeps the two apart: performance metrics are quantitative thresholds, and AI-evaluated conditions carry the qualitative checks. How to test and release voice agent changes covers where each one runs.
How are judge panels used to train a voice model?
Tell-1, Retell's own LLM, used LLM judges in two of its three training stages, according to the Retell LLM technical post. The preference-tuning stage learned from "preference pairs ranked by a panel of LLM judges".
In the reinforcement learning stage, the post says, "the model writes eight responses and every one is scored". Code runs first: calls to tools outside the agent's toolbox, malformed output and other critical errors "are caught in code before any judge sees it". For correctness, "multiple judge models grade tool-call correctness, hallucination and instruction following, and their verdicts are pooled into a single reward signal". For naturalness, a single judge ranks all eight responses side by side against a rubric, because "naturalness is easier to compare than to score".
That is the pattern this page describes: code checks for what can be verified, a judge panel for correctness, and ranking for tone.
Frequently asked questions about LLM as a judge
What does an LLM judge do?
It reads an output, such as a call transcript, along with grading instructions, and returns a verdict: pass or fail, a score, or a preference between two options.
How reliable is LLM as a judge?
It depends on the judge model and the task. One study found over 80% agreement with human preferences; another found that only the largest judges aligned reasonably with humans. Check a judge against hand-graded calls first.
What is a judge panel?
A judge panel is several judge models grading the same output, with their verdicts pooled. It reduces the effect of any one model's bias.
What is the difference between rubric grading and pairwise ranking?
Rubric grading scores one output against written criteria. Pairwise ranking shows the judge two outputs and asks which is better.
Can an LLM judge score a voice call?
Yes, from the transcript and tool calls. Qualities that live in the audio need the audio as well: Retell's docs say Agent Naturalness is "evaluated using both the audio and the transcript".
Score your own calls before you scale
Retell AI is an AI voice agent platform for contact centers. AI QA evaluates a sample of your calls against the conditions and thresholds you set, with transcript evidence for each result.
Security & compliance for regulated deployments
How Retell handles audits, data retention and access control.
Further Reading
Related terms
Table of contents
