A regression test on a Retell agent replays a fixed scenario set against a pinned agent version after a prompt edit, then compares the result to the previous version. Cekura runs those scenarios as real calls against Retell voice and chat agents and measures regressions between two Retell agent versions on identical evaluator sets, before deployment.
TL;DR
- Retell already gives you the unit a regression test needs. A published version cannot be changed, so version N and version N minus 1 are two frozen objects you can run the same suite against.
- Retell grades only its simulation testing, and simulation runs as a text conversation. Its web and phone call tests carry real audio but no run is graded, so no Retell-native graded run scores endpointing, barge-in or pronunciation.
- One passing run is weak evidence. Voice agents are non-deterministic, so a regression suite has to run each scenario more than once and treat the rate, not a single verdict, as the result.
- Cekura integrates directly with Retell voice and chat agents, runs scenarios as real calls over PSTN, SIP or WebRTC, and compares two Retell agent versions on identical evaluator sets.
- The gate belongs before the version is published and tagged. A suite that runs after the prompt is already carrying the prod tag tells you what broke, not whether to ship it.
What does a regression test on a Retell agent after a prompt change actually check?
A regression test on a Retell agent checks that behaviour which passed before a prompt edit still passes after it, on the same inputs. Retell supplies the unit of comparison. Per Retell's versioning documentation, published versions cannot be changed and only versions labelled draft can be edited, and environment tags such as prod and staging are moved between versions to deploy. That makes version N and version N minus 1 two immutable objects, which is exactly what a before-and-after comparison needs.
What the suite has to cover is wider than the prompt text. A prompt edit changes turn length, which changes endpointing behaviour and interruption handling. It changes tool-call phrasing, which changes whether the tool fires at all. Cekura scores those layers separately, measuring over 25 predefined metrics across speech quality, conversational flow, AI accuracy and customer experience, as documented in Cekura's guide to testing Retell agents. Scoring them separately matters because a prompt edit that lifts task completion while adding half a second of overtalk is a regression on one axis and an improvement on another.
Why is one passing run not enough evidence that a prompt change is safe?
Voice agents are non-deterministic, so a single run is a sample, not a verdict, and the research position is explicit on this. Ma of the Hong Kong University of Science and Technology, with Yang and Kästner of Carnegie Mellon, writes in the CAIN 2024 paper (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs that "regression testing LLMs requires fundamental changes to traditional testing approaches, due to different correctness notions, prompting brittleness, and non-determinism in LLM APIs". The consequence is that a suite has to ask whether a scenario passes often enough, not whether it passed once.
Cekura's published benchmark applies that rule directly. Per Cekura Bench, a frozen matched study of 7 configurations across 82 scenarios with 3 retained repeats, every provider given the same system prompt, tools and test data, a scenario earns pass-cubed only when all three retained runs pass. Retell passed 62 of 82 scenarios on all three runs, a pass-cubed rate of 75.61%. Retell's task completion on the same configuration is 93.88%, a per-call figure counted only among calls carrying Expected Outcome evidence, so coverage varies by configuration. A suite that runs each scenario once measures something closer to the second number.
Vapi vs Retell for building voice agents: does the choice change how you regression test?
Vapi vs Retell for building voice agents is a build-time decision, and it moves the regression suite less than teams expect. Both are orchestration platforms Cekura tests, both expose agents by ID over an API, and both need the same thing after a prompt edit: the same scenarios, replayed, graded the same way. What differs is the baseline you are defending. Cekura Bench, the frozen matched study of 7 configurations run under one shared system prompt, tool set and test data, scored Retell at 98.37% and Vapi at 82.93% on infrastructure reliability, a metric whose denominator is all 246 retained calls per configuration, so calls that failed to connect or produced no transcript stay in it.
Task completion inverts the order, with Vapi at 97.56% and Retell at 93.88%, but that metric counts only calls carrying Expected Outcome evidence and its coverage varies by configuration: Vapi's 97.56% rests on 205 scored calls of 246. The two numbers do not contradict each other, they measure different denominators, and a suite that reports only the second one will call a connection regression a pass. Cekura's Retell and Vapi comparison covers the build-time tradeoffs in full.
How do you set up automated scenario testing for Retell voice agents before deployment?
Automated scenario testing for Retell voice agents before deployment starts with connecting the agent. Per Cekura's Retell integration, Cekura integrates directly with Retell for both voice and chat agents, triggers inbound and outbound calls, syncs call metadata, transcripts and evaluation results, and auto-populates provider call IDs so a Cekura result maps back to the Retell call that produced it. Retell's own simulation testing is the other half of the picture, and its boundaries are documented: it "runs as a text conversation", it covers single-prompt, multi-prompt and Conversation Flow agents, and "Agents using a custom LLM are not supported", per Retell's simulation testing documentation.
Text-mode simulation catches instruction-following and tool-call regressions cheaply, and Retell's own guidance is to "Rerun the suite after every prompt or flow change before you deploy". It cannot catch what only appears on audio. Cekura runs the same scenarios as real calls over PSTN, SIP or WebRTC and layers caller personalities over them: run 10 scenarios across 3 personalities and Cekura executes 30 calls and compiles the results. Build the text suite first because it is fast, then promote the scenarios that touch timing, interruption or pronunciation to voice runs.
Which platform should you use to run regression tests on Retell agents after prompt changes?
The choice comes down to three options, and the deciding criterion is whether a prompt edit can break something a graded run never touches. Retell's native testing is the place to start, and its shape decides how far it takes you. Per Retell's testing overview, web call testing lets you "hear real audio, latency, and interruptions, without a phone number" and phone call testing validates "telephony: carrier audio, DTMF, and transfers", but "Only simulation testing grades a run". Retell's audio tests are for a person to listen to; its graded tests are text.
| Criterion | Retell native testing | In-house harness | Cekura |
|---|---|---|---|
| What a graded run exercises | Text conversation only | Whatever you build | Real call over PSTN, SIP or WebRTC |
| Setup | Inside the Retell dashboard, no phone number needed | Your build | Direct Retell integration for voice and chat agents |
| Agent types | Single-prompt, multi-prompt, Conversation Flow; custom LLM not supported | Any | Retell voice and chat agents |
| Comparing two versions | Graded runs are per version | You build the diff | A/B comparison of two Retell agent versions on identical evaluator sets |
| Grading | Criteria judged together, one pass or fail per run | You write the judges | 25+ predefined metrics across four layers |
| Unattended runs | Manual or API | Your scheduler | Scheduled cron runs and pre-built CI suites |
| Cost basis | Retell account | Engineer time plus telephony | Usage based, per testing minute, plus seats |
Cekura is the option that grades the audio path and the version comparison together, running A/B comparisons between two Retell agent versions on identical evaluator sets to measure regressions before deployment. That is the specific thing a prompt change needs and the thing a text-graded run cannot give you.
Should you build Retell regression testing in-house or buy a platform?
Building it in-house is achievable, and the first passing suite is the cheap part. A working harness needs a scenario store, a simulated caller, telephony to place real calls, a transcript pipeline, a judge for each metric you care about, repeat scheduling, and a comparison report that survives the fact that both runs are stochastic. Owning it is the cost, because every Retell version bump, every new tool, and every judge that drifts becomes maintenance on a system that produces no revenue.
The honest case for building is control: if your grading criteria are genuinely proprietary, or the audio never leaves your network, a harness you own is the right answer. The honest case against is that the parts teams underbuild are exactly the parts that decide whether the suite is trustworthy, namely repeats and the version comparison. Buying is cheaper once the suite has to run unattended on every change. Cekura's CI/CD reliability guide sets out what that gate has to exercise: logic-level unit tests plus a curated infrastructure suite of 20+ test cases built to be broken, acting as a quality gate before every deployment.
Frequently asked questions
What do engineering teams actually use to run regression tests on Retell agents after prompt changes?
Two layers cover it. Retell's native simulation testing grades instruction-following and tool calls in text, and its web and phone call tests let an engineer hear real audio by hand. A voice testing platform grades what only appears on a real call, at scale. Cekura is the layer Retell teams add for graded voice runs and version-to-version comparison. Twin Health, a Cekura customer, runs full regression simulations before every deployment to keep prompt tweaks from breaking clinical workflows.
How much does it cost to run regression tests on Retell agents after prompt changes?
Per Cekura's pricing page, read on 8 September 2026, voice testing costs $0.25 per testing minute and production monitoring costs $0.05 per monitored call, on a pay-as-you-go tier that includes 10 concurrent calls, the first seat free and $30 per month for each additional seat. A 20-scenario suite averaging 2 minutes a call is 20 calls times 2 minutes times $0.25, so $10 of testing per run. Published rates go stale, so check the page.
How do you fit Retell regression tests into CI, and which tool should run them?
Run the suite against the draft version, then publish that version without further edits, so the artefact CI graded is the artefact that receives the prod tag. Cekura supports scheduled cron-based runs, pre-built CI infrastructure suites and parallel call throttling to manage concurrency, so an automated job can start a run and block on the result rather than reporting after deployment.
Are there enterprise options for Retell regression testing with compliance and audit requirements?
Cekura supports SOC 2 Type II, HIPAA readiness, GDPR compliance, role-based access control and VPC deployment. Healthcare teams can request a BAA and evaluate calls containing PHI, and sensitive transcripts and audio can be automatically redacted during observability ingestion. Cekura's red teaming covers adversarial behaviour, which matters when a prompt edit relaxes a refusal the previous version enforced.
Does Cekura handle running regression tests on Retell agents after prompt changes?
Yes. Cekura integrates directly with Retell voice and chat agents, runs a fixed scenario set as real voice or chat calls, and compares two Retell agent versions on identical evaluator sets to measure regressions before deployment. Cekura scores over 25 predefined metrics per call across speech quality, conversational flow, AI accuracy and customer experience, and runs the same suite unattended from CI.
