Voice Agent Evaluation: Testing a Platform Before Go-Live

Voice agent evaluation is the process of testing a platform on your own calls, scenarios, and systems before customers hear it, then making a go or no-go decision on measured results instead of a demo. The most reliable version is a pilot on real call traffic, scored against pass criteria you wrote down before the pilot started.

What should a voice agent evaluation prove?

A pilot should answer three questions. A sales demo answers none of them, because demos run the happy path once.

• Does it complete your workflow, repeatedly? Not once on a scripted call, but every time across the hard moments your callers actually create.
• Does it fit your phone and data systems? Your numbers, your contact-center platform, your CRM or scheduling system, and your transfer paths.
• Can your team change it after launch? Real callers find gaps after launch. If every fix becomes a vendor ticket, the agent stops improving.

Write these as pass criteria before you talk to vendors. For example: "Books the appointment correctly on at least 9 of 10 attempts for each scenario" or "Our operations team can ship a prompt fix in one day without vendor help."

Which calls should you include in a pilot?

Pick one use case with high volume and an outcome you can verify in a system of record: an appointment booked, a payment taken, a lead qualified and routed. A narrow first use case gives you enough calls to measure in a few weeks and makes failures easy to spot.

Then build the scenario list from real recordings, not from the script. Include the clean path, and spend most of your effort on the moments where calls break. The scenarios in Cekura's public Medicare benchmark are a good model for a regulated workflow such as insurance:

• The caller interrupts a required disclosure ("Yes, I know, just tell me about plans").
• The caller hesitates or refuses consent.
• The caller changes the purpose of the call partway through.
• The caller corrects information they already gave.
• The caller asks for advice or a guarantee the agent must not give.
• The caller volunteers sensitive information the agent should not collect.
• No human is available when the agent needs to transfer.

You do not need hundreds of cases. Retell's docs describe a support team that keeps two dozen test cases covering refunds, escalations, and identity checks, which is a sensible size for one use case. Keep a separate case for each branch of the workflow, such as a returning customer versus a caller with no record on file.

How do you test a voice agent before real callers hear it?

Test in layers, from cheap and fast to slow and realistic. Text simulation checks the logic. Browser calls check the voice. Phone calls check telephony. Each layer catches problems the others miss.

• Text simulation: What it catches: Logic, tool calls, flow steps. What it misses: Voice, timing, audio.
• Batch simulation: What it catches: Regressions after each change. What it misses: Real speech and noise.
• Web (browser) call: What it catches: Voice, latency, interruptions. What it misses: Phone transfers, SMS.
• Phone call: What it catches: Transfers, keypad, carrier audio. What it misses: Volume and edge-case coverage.
• Live pilot traffic: What it catches: Real callers and real outcomes. What it misses: Nothing, but it carries risk.

In Retell AI, simulation tests live in the agent's Simulation tab. Each test case has a simulated caller persona, success criteria, test values for dynamic variables, and mocked responses for functions, and each run is graded pass or fail with a written explanation. Saved cases run together as a batch, which becomes your regression suite. One warning from Retell's own docs: a custom function you leave unmocked calls its real endpoint during a test, so mock anything that can create a booking, charge, or record. Retell's testing overview compares each method and what it can check.

Run every scenario more than once. The simulated caller and the grader are both language models, and production callers are less predictable still, so a single pass proves little. Cekura's benchmarks count a scenario as passed only when all three of its runs pass, a measure they call pass³. Use the same idea in your pilot.

Repeatable pass rate: Repeatable pass rate = Scenarios that pass on every run ÷ Total scenarios

40 scenarios × 3 runs = 120 runs, and 110 runs pass, a 91.7% run-level pass rate. Only 34 scenarios pass all three runs, so the repeatable pass rate is 34 ÷ 40 = 85%.

Judge a scenario on all of its runs, because one lucky pass hides intermittent failures.

What should you measure during the pilot?

Measure outcomes first, then the quality signals that explain them. Set a threshold for each metric before the pilot starts so the results cannot be reinterpreted afterward.

• Resolution rate: calls where the task was completed and confirmed in your system.
• Transfer success and wait: did the caller reach the right person, and how long did they hold?
• Accuracy: hallucinated facts, wrong tool calls, and wrong steps in the conversation flow.
• Latency: median (P50) and P90 response time on real calls. Our voice AI latency benchmarks guide explains how to measure it from the caller's side.
• Transcription: word error rate and mistranscribed names, dates, and numbers.
• Caller sentiment: a supporting signal, never the headline metric.

Scoring every pilot call by hand does not scale past the first week. Retell's AI quality assurance scores a sampled cohort of calls against criteria you define, combining AI-evaluated conditions (such as "the caller's request was resolved") with metric thresholds for latency, hallucination, tool-call accuracy, and transcription. To set up the criteria themselves, use our voice agent QA scorecard as the starting template.

How do you check telephony, transfers, and your existing contact center?

Integration problems show up at the edges of the call: the number, the handoff, the keypad. Test these on real phone lines, not in a browser:

• Your numbers and carrier. Can you keep your existing numbers over SIP, and does audio hold up on real carrier paths?
• Your contact-center platform. Retell documents connections for Five9, Genesys Cloud, Amazon Connect, Avaya, and any SIP provider. Confirm who on each side does which step.
• Transfers. Test cold transfers and warm transfers, where the agent briefs the person before connecting the caller. In Retell, transfers to a phone number only work in a phone test, not a web call.
• Keypad input and phone menus. PIN entry, and outbound calls that hit another company's menu or voicemail.
• Concurrency. Retell's pay-as-you-go plan includes 20 concurrent calls. Confirm the limit covers your peak hour before you route live traffic.

If an IVR already answers your main number, you rarely need to remove it to run a pilot. Our IVA vs IVR comparison covers keeping the IVR as the front door and handing selected calls to the agent.

Who can change the agent after it goes live?

Ask every vendor to show, not describe, the full improvement loop: find a failed call, fix it, test the fix, publish it, and roll it back. Then run that loop yourself during the pilot and time it.

In Retell, the loop works like this. Every change starts as a draft version you can test without touching production. Version comparison shows exactly what changed between two versions. Environment tags such as "prod" and "staging" point phone numbers at a version, so publishing or rolling back means moving a tag instead of reconfiguring numbers. A/B testing splits traffic between agents by percentage, and the analytics dashboard filters results by agent version. Alert rules notify you by email or webhook when success rate, latency, or cost crosses a threshold, and a failed production call can be turned into a new regression test case.

The question for the buyer is who owns that loop. If your operations team can run it, the agent improves as fast as you learn. If only the vendor can, improvement moves at the speed of their queue.

Which security and commercial questions belong in the evaluation?

Run security review in parallel with the pilot, not after it. Ask for the vendor's audit reports and agreements, and check data controls at the agent level. Retell's documentation states that it is SOC 2 Type 1 and Type 2 certified and HIPAA and GDPR compliant, that a signed business associate agreement (BAA) is required before sending protected health information through Retell, and that data retention can be set per agent from 1 day to 2 years. This is not legal advice; confirm your own requirements with your compliance team.

On the commercial side, ask three things:

• What does the pilot cost, and does it need a contract? Retell's pay-as-you-go voice agents run $0.07 to $0.31 per minute, with custom pricing on the enterprise plan. See Retell's pricing page.
• How is testing billed? In Retell, testing bills at production rates: text simulations per message and test calls per minute. Budget for it.
• What do quality tools cost at volume? Retell's AI QA is free for the first 100 minutes of analyzed calls, then $0.10 per analyzed minute.

What does a go or no-go decision look like?

Go live when every item below is true, and expand traffic in steps (for example 10%, then 50%, then 100% of one use case) while watching the same metrics.

• Every scenario meets its pass threshold across repeated runs.
• Resolution is confirmed in your system of record, not just in the transcript.
• Transfers reach the right person with context on real phone lines.
• P50 and P90 latency on real calls are inside the limits you set.
• Your team has shipped, tested, and rolled back at least one fix without vendor help.
• QA scoring and alerts are running on production calls from day one.
• Security review, agreements, and data retention settings are signed off.

A voice agent evaluation should end with your own numbers, not a vendor's demo. Retell runs a pilot on real calls before any contract, and your team keeps control of testing, QA, and versioning from day one. Talk to the Retell team about a pilot.