QA Scorecard Template for Voice Agent Call Quality

A voice agent QA scorecard is a fixed set of criteria, weights, and pass rules that every reviewed call is graded against, so call quality is measured the same way no matter who, or what, does the reviewing. This QA scorecard template is built for AI voice agents. It keeps the parts of a human-agent scorecard that still apply and adds the checks AI agents need, such as tool use, factual accuracy, and response time.

What goes on a voice agent QA scorecard?

Use the eight criteria below as your starting template. Weights add up to 100, and you should adjust them to match what a failed call costs your business.

• Task resolved: Pass looks like: Outcome confirmed in your system. Weight: 25.
• Accuracy: Pass looks like: No invented or wrong facts. Weight: 20.
• Tool and data use: Pass looks like: Right tool, right inputs. Weight: 15.
• Required steps: Pass looks like: Verification and disclosures done. Weight: 15.
• Escalation: Pass looks like: Right transfer, with context. Weight: 10.
• Responsiveness: Pass looks like: Median delay under your limit. Weight: 5.
• Turn-taking: Pass looks like: Yields when the caller interrupts. Weight: 5.
• Tone and clarity: Pass looks like: Clear, concise, on-brand. Weight: 5.

Define each criterion in terms your reviewers can check from the recording, transcript, and system logs:

• Task resolved means the outcome exists in your system of record. For a lead qualification agent, that is a qualified lead routed to the right team with the required fields filled, not a transcript where the agent said "I've passed that along."
• Accuracy means every fact the agent stated is supported by its instructions, knowledge base, or the caller. Invented case numbers, wrong hours, and contradictions all fail.
• Tool and data use means the agent called the right function with the right inputs. Check the tool call itself, not just what the agent said about it.
• Required steps covers anything mandatory: identity verification, a recording notice, a disclosure. This row is pass or fail, and a fail usually fails the whole call.
• Escalation means the agent transferred when it should have, reached the right destination, and passed context so the caller did not repeat themselves.
• Responsiveness, turn-taking, and tone cover how the call felt: response delay, talking over the caller, and clear, concise wording.

How do you score a call with the template?

Score each criterion pass or fail, add up the weights of the criteria that passed, then apply your critical-error rule.

Weighted QA score: QA score = (Sum of weights for passed criteria ÷ Sum of all weights) × 100

A call passes everything except tool use (15) and turn-taking (5): (100 − 20) ÷ 100 × 100 = 80. The same call skips a required disclosure, a critical error, so it scores 0 and fails.

Apply critical errors after the math, because an auto-fail sets the score to zero.

Critical errors override the weighted score. Decide your list before you score the first call, and treat any of these as an automatic fail:

• Skipped identity verification or a required disclosure.
• Stated a price, policy, or eligibility the agent was not given.
• Took a real action (booking, payment, update) with the wrong details.
• Collected sensitive data the workflow does not need.
• Refused or ignored a clear request for a person.
• Ended the call before the caller's request was handled.

How is scoring an AI agent different from scoring a human agent?

Human QA exists to coach individuals. AI QA exists to find patterns and fix them once. That changes how you use the scorecard in four ways.

• Failures repeat. An AI agent that mishandles a request on one call will usually mishandle it on similar calls. Score enough calls to see the pattern, then fix the cause.
• The fix is a change, not a conversation. A failed criterion leads to a prompt, flow, or knowledge-base edit, followed by a regression test, rather than a coaching session.
• The transcript is not enough. You also need tool calls and flow steps. In one run of Cekura's public voice agent benchmark, the transcript showed the caller's phone number correctly, but the agent sent a different number to the tool. A transcript-only review would have passed that call.
• Timing is measurable. Response delay and overlapping speech can be read from call data instead of a reviewer's impression. Our guide to voice AI latency benchmarks explains which percentile to set limits on.

How many calls should you review?

There is no industry-standard sample size for AI agents, so set it by risk. A practical pattern is to review every call that fails an automated check, plus a random sample of calls that passed, so you catch errors the automation missed. Review more heavily in the first weeks after launch and after every significant change.

Keep a person in the loop for calibration. Each week, have a reviewer grade a small set of calls by hand and compare the results with the automated scores. Where they disagree, either the criterion wording is ambiguous or the threshold is wrong. Fix the definition, not just the score. The same scorecard also works on human-handled calls, which is the basis of most call quality monitoring programs, so you can compare AI and human calls on equal terms.

How do you automate this scorecard in Retell?

Retell AI's AI QA maps closely to this template. You create a cohort of calls (filtered by agent, date range, call length, and other fields), set a sampling percentage and a weekly maximum, and then define resolution criteria in two parts:

• AI-evaluated conditions for the judgment rows: task resolved, required steps completed, escalation handled correctly. Each is a named check with a prompt that describes success.
• Performance metrics with thresholds for the measurable rows: latency, agent hallucination, tool call inaccuracy, node transition inaccuracy, word error rate, overlapping speech, agent naturalness, and sentiment.

Turn on weighted scoring to assign the weights from the template and set the score a call must reach to pass. Reviewers can calibrate an individual call to success or failure, which changes that call's score without changing the criteria. The AI QA metrics reference defines how each metric is calculated. As one reference point, Retell's QA troubleshooting guide treats a median end-to-end latency above 2.5 seconds as high.

For outcome fields you need on every call, Post Call Extraction writes structured results such as a call summary, a success flag, and custom fields onto each call, and those fields can filter a QA cohort. Alert rules can notify you by email or webhook when the number of calls failing QA rises.

What should you do when a call fails the scorecard?

Treat a failure as the start of a fix, not the end of a review.

1. Review several failed calls on the same criterion to confirm the pattern before changing anything.
2. Make the change in a draft version of the agent.
3. Turn the failed call into a test case and rerun your regression batch.
4. Publish the version, then watch the criterion's pass rate on new calls.

Track the scorecard's pass rate per criterion over time on a dashboard. Our voice agent KPI dashboard template shows how to lay it out. If you are still choosing a platform, use this scorecard as the grading sheet for your pilot, as described in our guide to voice agent evaluation before go-live.

A scorecard only improves quality if your team can act on what it finds. Retell lets your own team run QA, testing, and versioning on its agents, and you can prove it on a pilot with real calls before any contract. Talk to the Retell team about a pilot.