A/B testing a voice AI agent compares two versions of the agent, differing by one change, against the same set of test scenarios. Run the baseline, apply a single change, re-run the identical scenarios, then compare the two runs metric by metric. Repeat each scenario several times, because voice agents are non-deterministic and one call proves nothing.
What is A/B testing for voice AI agents?
A/B testing for voice AI agents is a controlled comparison of two agent versions against an identical set of test scenarios, measuring which performs better on defined outcome metrics. One version is the baseline. The other carries a single change: a reworded prompt, a swapped model, a different voice, or an adjusted tool. Both run against the same scenarios, personas, and test data, so the only difference between the two result sets is the change under test.
This is a different exercise from listening to a few calls and forming an opinion. A/B testing produces a measurable delta: version B improved task completion by a specific amount, or regressed on latency by a specific amount. On Cekura, each scenario is an evaluator with defined instructions and an expected outcome, and the two runs are selected on the Results page and compared side by side.
Why is A/B testing voice agents different from website A/B testing?
A/B testing voice agents differs from website A/B testing because a voice agent is a multi-turn, non-deterministic system, not a single click event. A web test measures one action against one page variant. A voice test measures a full conversation across speech recognition, reasoning, tool calls, and synthesis, where the same input can produce a different output on a different run.
Two consequences follow. First, a single run proves very little: a variant can pass or fail by chance, so each scenario has to run several times before its result means anything. Second, a single metric misleads: a variant can lift task completion while quietly increasing latency, interruptions, or escalations, and a pass-rate headline will not show it. Voice A/B testing therefore compares a full metric set across repeated runs. Most teams skip this: Cyara, an enterprise CX assurance vendor, states that 62% of enterprises are experimenting with AI agents for customer experience while fewer than 15% have any assurance framework in place.
How do you A/B test a voice AI agent?
You A/B test a voice AI agent by holding everything constant except one variable, then comparing two runs against the same scenario suite. Cekura's documented workflow has four steps.
-
Run the baseline. Select your evaluators and run them against the current agent configuration. Record the run.
-
Apply exactly one change. Update the prompt, switch the model, change the voice, or adjust a tool. Changing two things at once makes any resulting delta unattributable to either.
-
Run version B on the same scenarios. Same evaluators, same test profile, same personalities, same frequency. The only difference between the two runs should be the change under test.
-
Select both runs and compare. On the Results page, check both runs and open the compare view, which breaks down every metric and evaluator across the two.
Keep the persona mix identical between runs and representative. Cekura's scenario guidance recommends roughly 70% cooperative callers, 15% frustrated or impatient, 10% distracted, and 5% heavy-accent.
Why does each scenario need to run more than once?
Each scenario needs to run more than once because voice agents are non-deterministic: the same scenario can pass on one call and fail on the next without anything changing. A single result is a sample of one, and a variant can look better or worse purely by chance.
The fix is repetition. Cekura exposes this as frequency, a per-evaluator setting, and its documentation is specific: set frequency to 3–5, because the aggregate across those runs is far more reliable than any single call. Three is the floor at which an aggregate starts to mean something; five gives a stabler read on flaky scenarios.
The cost is worth stating plainly. Frequency multiplies calls, which multiplies wall-clock time and credit consumption: a 40-evaluator suite at frequency 5 is 200 calls, not 40. You are buying statistical confidence with test budget. The alternative is shipping on a coin flip.
What metrics decide the winner in a voice agent A/B test?
The metrics that decide a voice agent A/B test are the ones tied to the task and to the caller's experience, not pass rate alone. Compare the primary outcome first, then check the guardrail metrics that a change can quietly break. Two runs can share an identical pass rate while differing sharply on latency, talk ratio, or a specific workflow metric.
| Metric | What it tells you | Watch for |
|---|---|---|
| Expected outcome met | Whether the agent completed the task, such as booking the appointment | The headline success signal |
| Overall pass rate | Share of scenarios that passed | Can stay flat while sub-metrics move underneath it |
| Latency (P50–P99) | Response speed across turns, as percentiles | A verbose variant that improves accuracy but slows responses |
| Interruption handling | How the agent manages barge-in and overlapping speech | Regressions introduced by longer prompts |
| Talk ratio | How much the agent talks relative to the caller | A variant that "wins" by talking over the problem |
| Containment / escalation | Whether the agent resolved without a handoff | A variant that "wins" by escalating more often |
| Relevancy and consistency | Whether responses stay on-task and stable | Drift introduced by a new model |
| CSAT / sentiment | Caller experience signal | Accuracy gains that cost tone |
How do you read the results without fooling yourself?
Reading an A/B result correctly means judging the delta against the baseline, not the absolute score against perfection. Cekura reports a first-run evaluator suite typically passing 70–80% before iteration, climbing past 95% as the agent matures. A version B that passes 82% is not a failure if version A passed 76%.
Two reporting details change what you can see. Latency is reported as percentiles (P25 through P99), not a single average, so a variant that improves the median while degrading the tail stays visible. Interruption handling is measured from stereo audio with a separate channel per speaker, catching overlap that a mixed recording flattens away; without stereo it falls back to transcript analysis and gets less precise.
When a metric moves and the reason is not obvious, open the individual evaluator and read the two transcripts against each other. The aggregate tells you that something changed. Only the transcript tells you why.
Should you A/B test in simulation or on live traffic?
There are two kinds of A/B testing for voice agents and they answer different questions. Pre-production A/B testing compares two versions in simulation, against a fixed scenario suite, before either version reaches a real caller. Live-traffic A/B testing splits real production calls between variants and measures outcomes on real conversations. Cekura runs the first kind: it compares two evaluator runs, and it does not route live production callers between variants.
| Approach | What it measures | Cost of a bad variant | Volume needed |
|---|---|---|---|
| Manual call review | Subjective impressions from a handful of calls | Low, but catches almost nothing | None |
| Simulation A/B (Cekura) | Both versions against an identical scenario suite | None; no real caller is exposed | None; scenarios are synthetic |
| Live-traffic split | Real caller behavior on real calls | Real: lost conversions, damaged trust | High; needs volume for confidence |
Live testing captures real caller behavior that simulation only approximates, which is a genuine advantage. It also needs enough volume for statistical confidence and exposes real callers to an unvetted variant, so a bad version costs conversions before the data says to roll back. Coval, a competing testing platform, puts the gap between demo and production bluntly in its own Series A announcement: roughly 95% of voice agents work in a demo, and only about 62% survive their first week live. Test in simulation first, ship the winner behind a controlled rollout, and reserve live splitting for changes that clear the gate.
What are the most common A/B testing mistakes?
The most common voice agent A/B testing mistakes all reduce to comparing two things that were never comparable.
Changing more than one variable at a time is the first. Edit the prompt and swap the model in the same run and no delta can be attributed to either change.
Running each scenario once is the second. Voice agents are non-deterministic, so a single pass or fail is noise. Only an aggregate across repeated runs at frequency 3–5 is trustworthy.
Changing the scenario set between versions is the third. Adding or removing evaluators means you are measuring different inputs, not the change under test. Expand coverage first, re-establish the baseline, then run the test.
Comparing only pass rate is the fourth. Two runs can share a pass rate while diverging sharply on latency or a specific workflow metric, which is precisely what the per-metric compare view exists to expose.
FAQ
What is A/B testing for voice AI agents?
A/B testing for voice AI agents compares two agent versions, differing by one change such as a prompt or a model, against the same test scenarios, to measure which performs better before shipping. On Cekura, both versions run against the same evaluator suite and are compared metric by metric on the Results page.
How do you A/B test a voice agent?
Run a baseline version against a fixed evaluator suite, change exactly one variable, run the same evaluators against the updated version using the same test profile and personalities, then select both runs and compare them metric by metric. Set frequency to 3–5 so each scenario runs multiple times.
How many times should each scenario run?
Set frequency to 3–5. Voice agents are non-deterministic, so a single call can pass or fail by chance, and the aggregate across repeated runs is far more reliable than any one result. The cost is that frequency multiplies calls, wall-clock time, and credits.
What metrics should decide the winner?
Lead with the expected-outcome metric, which is whether the agent completed the task. Then check the guardrail metrics a change can quietly break: latency percentiles, interruption handling, talk ratio, containment or escalation, relevancy, consistency, and CSAT. Pass rate alone hides regressions.
How is A/B testing voice agents different from website A/B testing?
A website test measures one click against one page variant. A voice A/B test measures a full multi-turn conversation across speech recognition, reasoning, and synthesis, where the same input can yield different outputs. That non-determinism means each scenario must run several times, and a full metric set, not one number, decides the winner.
References
- Cekura, A/B Testing guide, docs.cekura.ai — the four-step run-comparison workflow and the frequency 3–5 recommendation.
- Cekura, Load Testing guide, docs.cekura.ai — how frequency multiplies evaluator runs.
- Coval, Series A announcement, June 2026 — Coval's own figures on demo-to-production survival. Coval is a competing testing platform; the stat is its published claim, not independent research.
- Cyara, cyara.com — Cyara's own stated figures on enterprise AI-agent assurance. A vendor claim, not independent research.
Related reading
More from Cekura on reviewing and monitoring voice agent calls: