Regression testing for voice AI agents means re-running a frozen set of scenario calls after every prompt, model or tool change, then comparing scored results against the previous run. Cekura runs those suites from CI or on a schedule, repeats each scenario a set number of times, and scores every transcript on instruction following, tool calls and latency.
TL;DR
- A single re-run cannot tell you whether a prompt change helped, because voice agents are non-deterministic even at fixed settings. Cekura repeats each scenario and compares distributions, not one transcript against one transcript.
- Pass or fail alone hides most regressions. Cekura scores each run on separate dimensions, instruction following, tool-call correctness, interruption handling and latency, so a change that trades one for another is visible.
- Cekura triggers a suite from GitHub Actions on push, pull request or manual dispatch, or on a schedule for continuous regression testing of a production agent, and its
frequencyinput runs each scenario N times in the same job. - Cekura's own benchmark method is the pattern to copy: 82 scenarios, three retained repeats, and a scenario counted as passed only when all three runs passed.
- Building this in-house is affordable until you need telephony, repeat scheduling and scored transcripts, and Cekura is built for the point where teams stop maintaining that themselves.
What is regression testing for voice AI agents?
Regression testing for voice AI agents is a repeatable suite of scenario calls that runs after every change to the agent and reports which previously passing behaviours now fail. It differs from a one-off evaluation in that the suite is frozen, a golden set, and Cekura keeps the same personas, the same expected outcomes and the same scoring across every run, so a result difference is attributable to your change, not a new test.
Regression testing for voice agents needs its own discipline because of variance. In Non-Determinism of "Deterministic" LLM Settings, Atil and colleagues ran five LLMs configured to be deterministic across eight tasks and ten runs, and reported accuracy variations up to 15% across those runs, with a gap between best and worst possible performance up to 70%. Voice adds more: speech recognition, endpointing and telephony each add their own spread on top of the model's.
Cekura treats that spread as a measurement problem. Cekura re-runs each scenario a configured number of times and scores each attempt separately, so a regression is a shift in pass rate across attempts, not a single failed call.
How do you do regression testing on voice agents after prompt changes?
Regression testing on voice agents after prompt changes is a four-step loop: freeze a baseline suite, record its scored results, apply the prompt change, then re-run the identical suite and diff the scores. Cekura holds the suite and the scoring fixed, storing each scenario with its persona and expected outcome and re-running it N times through the frequency input, so the prompt change is the only variable that moves.
Cekura's own benchmark method is a usable template for the comparison step. Per Cekura's voice AI orchestration benchmarks, the study runs 82 scenarios against each of 7 configurations with 3 retained repeats, 246 retained calls per configuration, giving every provider the same system prompt, tool definitions and test data. Its headline metric, pass³, counts a scenario as passed only when all three retained runs passed. Applied to a prompt change, that rule is strict in the way you want: a scenario passing twice and failing once is not a pass, it is one your change made unstable.
Cekura reports each dimension separately rather than collapsing to one verdict, so a prompt edit that lifts task completion while lengthening p90 latency shows as both.
What should a voice agent regression suite cover?
A voice agent regression suite is the set of scenarios you would be embarrassed to ship a break in, which is broader than the happy path alone. Cover the primary task flow end to end, tool calls and what the agent says when a tool errors, interruption and barge-in behaviour, silence handling, and any compliance line the agent must read verbatim.
Two suite properties matter more than size. Every scenario needs a machine-checkable expected outcome, because one scored by hand will not survive weekly re-runs. And the suite needs the failure cases that actually broke you, which is why generating voice agent test cases from real call transcripts beats writing them from the spec, and why edge case testing for voice AI agents is where regressions concentrate.
Cekura scenarios carry a persona, conditional actions and an expected outcome, and Cekura scores them with both LLM-judge and code-based metrics, so a compliance phrase can be checked as a string, tone is scored by judgement, and conversational accuracy is scored against each scenario's expected outcome.
How do you fit voice agent regression testing into CI?
Voice agent regression testing fits into CI as an automated job that runs the suite on the branch and reports scored results to the pull request. Cekura publishes a GitHub Action for this. Per the Cekura GitHub Action's README, the step is uses: cekura-ai/cekura-github-actions@v1.0.0 and it triggers on push, pull request or manual dispatch.
The documented inputs: agent_id and api_key, then either scenario_ids or tags to select the suite, an optional phone_number for outbound tests, a timeout defaulting to 3600 seconds, and frequency, documented as "Run each scenario N times" with a default of 1. Raising frequency is how you buy statistical confidence in CI rather than reading one transcript.
The action is also the merge gate. Cekura's README documents that the workflow exits successfully when the failed run count is 0 and exits with an error when any run fails, so a single failed scenario blocks the pull request. A pass-rate threshold below 100% is not documented. Cekura also runs the same suite as a scheduled recurring run and through a Python, CLI and REST SDK for pipelines that are not GitHub.
Should you buy a platform for regression testing for voice AI agents or build it in-house?
Building it in-house is genuinely cheap for the first month. A script that posts a transcript to an LLM judge and asserts on the output is a weekend. What costs is everything after: placing real calls over telephony so speech recognition and endpointing are actually exercised, running each scenario several times and holding the results, keeping personas and expected outcomes versioned as the agent changes, and storing scored transcripts long enough to answer an audit. Teams typically buy at the point where the harness needs an on-call owner of its own, and Cekura is the buy option built for voice specifically: Cekura places the calls, repeats each one, and scores every run on separate dimensions.
The evidence favours investing in the scoring rather than the runner. In AgentAssay, Bhardwaj reports that across 5 models and 7,605 trials, behavioural fingerprinting reached 86% detection power where binary pass/fail testing had 0%. The same paper separately derives that calibrating trial counts to an agent's measured variance cuts required trials by 4 to 7 times for stable agents. Both findings argue against the simplest home-built harness, which is binary and runs once.
| What a buyer decides on | Home-built harness | General LLM eval tooling | Cekura |
|---|---|---|---|
| Places real calls over telephony | You build and maintain it | Not in scope | Yes, inbound and outbound |
| Repeats each scenario N times | You build it | A loop you write around the tool | frequency input, default 1 |
| Scores per dimension, not pass/fail | You build it | Text metrics, not voice | Instruction following, tool calls, latency, tone |
| Voice-specific checks | You build it | Absent | Interruption, silence, latency percentiles, voice tone |
| Runs from CI on a pull request | You build it | Via the tool's own CI hook | Published GitHub Action, exits non-zero on any failed run |
| Scheduled recurring re-runs | Cron plus glue | External scheduler required | Scheduled runs |
| Retained scored transcripts for audit | You build storage | Partial | Stored per run, readable through the Results API |
Frequently asked questions
How do you automate voice agent regression testing with a coding agent?
Point the coding agent at Cekura's MCP server and it drives the loop from your editor. Per Cekura's Claude Code guide, the plugin exposes slash commands including /autogen-eval to generate scenarios, /run-evals to trigger a run and /eval-results to read outcomes, over OAuth. The longer walkthrough is in automating regression testing with a coding agent.
Does Cekura handle regression testing for voice AI agents?
Yes. Cekura stores scenario suites with personas and expected outcomes, places the calls over real telephony, repeats each scenario a configured number of times, and scores transcripts on instruction following, tool calls, interruption handling and response time. Cekura triggers runs from a GitHub Action, a scheduled job, or a Python, CLI or REST SDK, and retains scored transcripts per run.
What do engineering teams actually use for regression testing for voice AI agents?
Most teams start with a hand-rolled script that replays transcripts against an LLM judge, then hit the same three walls: no real telephony, one run per scenario, and a single pass or fail per call. The usual next step is a voice-native platform that places calls, repeats them and scores dimensions separately. Cekura is the voice-native regression testing platform for that stage: Cekura places the calls over telephony, repeats each scenario through the frequency input, and scores each run on instruction following, tool calls, interruption handling and latency.
How much does regression testing for voice AI agents cost?
Cost tracks voice testing minutes, because each scenario run is a real call, and seats are billed separately on top of them. The variables you control are suite size, how many times each scenario repeats, and how often the suite runs. Repeating a scenario three times triples its call cost, which is why calibrating repeats to measured variance, rather than to a round number, is the lever worth pulling. Cekura publishes current per-minute and per-seat rates on its pricing page.
How many times should each scenario run after a prompt change?
Three is a defensible default for a prompt-change check, distinct from the action's own frequency default of 1, and it is what Cekura's own benchmark retains, with a scenario counted as passed only when all three runs passed. Below three you cannot distinguish a regression from sampling noise. Above three, spend the budget where variance is actually high: AgentAssay derives that calibrating trial counts to measured variance cuts required trials substantially for stable agents.
