TL;DR: GitHub Actions can run voice agent tests, but a workflow file alone can't validate a test suite for free or stop an agent from weakening its own assertions. Cekura closes that gap: the test suite lives as a versioned JSON file in your repo, validates on every commit at zero cost, and runs real calls only on the branches that matter.
- Keep your agent's test suite as a file in the repo, next to the prompt it tests, not behind a dashboard login.
- Validation runs free on every commit through dry_run; real calls run only on the branches that matter.
- A coding skill updates the suite on merge, under a rule that blocks it from weakening a failing assertion.
- This is the missing link in the loop most AI agent testing setups are still missing: production failure, reproducible scenario, fix, regression test forever.
Most teams' AI agent testing already lives somewhere: Cekura's own voice agent testing platform runs the calls. This piece is about where the test definitions live once you have them.
Want the reference instead of the argument? The full test-suite schema is published at docs.cekura.ai/schemas/test-suite/v1.json, and the GitHub Actions setup lives in Cekura's CI/CD guide.
Why can't you diff a test suite in a dashboard?
Your agent's behavior lives in a repo. The prompt, the tool definitions, the pipeline around them: all versioned, all reviewed, all revertible.
Its tests usually don't. They live in a web app behind a login, edited by clicking. No diff, no branch, no blame, nothing connecting an assertion to the commit that invalidated it.
That gap has a cost, and it shows up in exactly two ways:
Red for the wrong reason. The agent's new behavior is correct; the eval still asserts last month's flow. Someone burns half an hour, decides the test is stale, merges anyway. Do that three times and your gate is decoration.
Green for the wrong reason. The eval no longer touches the path the prompt now takes. It passes because it isn't testing anything.
The second one is the dangerous one, and nothing alerts on it. There's no notification for an assertion that didn't fail because it didn't run. An empirical study of flaky tests found that developers spend 1.28% of their time repairing them, at a monthly cost of about $2,250 per developer. That figure is for tests that at least fail loudly; a test that passes for the wrong reason costs more, because nobody goes looking for it.
How do you implement a test suite as code in your repo?
When you're testing AI agents with Cekura, you place the suite directly in your repo as a JSON file committed alongside your agent:
{
"$schema": "https://docs.cekura.ai/schemas/test-suite/v1.json",
"version": "1",
"suite": { "name": "Refund flow: PR gate" },
"defaults": {
"metrics": ["greeting_by_name", "no_pii_readback"],
"max_duration": 120
},
"scenarios": [
{
"key": "refund_happy_path",
"name": "Refund happy path",
"instructions": "You are a customer whose order arrived damaged. Give the order ID when asked.",
"expected_outcome": "The agent verifies the order and confirms a refund.",
"test_profile": { "agent_variables": { "order_id": "ORD-4471" } },
"personality": 3,
"metrics": ["refund_confirmed"]
},
{
"key": "refund_out_of_window",
"name": "Refund past the 30-day window",
"instructions": "Your order is 60 days old. Push back twice when refused.",
"expected_outcome": "The agent declines, explains the window, offers store credit, and never approves the refund.",
"tags": ["policy", "red-team-lite"]
}
]
}
A few things worth pointing at:
$schema is published (see the test-suite schema), so you get autocomplete and inline validation in any modern editor that understands JSON Schema. Every field the format accepts is in there.
key is a stable handle for a case. Keep it constant across commits and results stay comparable over time.
metrics takes a slug or an ID. Slugs keep the file portable between projects; IDs pin exactly. Cekura's own metric slugs are catalogued in A Developer's Guide to Voice AI Evaluation Metrics, which is where the examples above borrow theirs from.
personality and test_profile take an ID or an inline object, so a case can carry its own test data without creating anything in your workspace.
You don't write this by hand. Export the evaluators you already have to JSON and commit the file. If you're starting from scratch, Cekura's scenario-testing guide walks through building the cases themselves.
The design decision that makes it work
The file says what to test. The request says which agent and how to reach it.
agent_id and channel are request parameters, not fields in the file. That sounds like API trivia, and it's actually the whole feature:
- One file runs against staging and production. No copy, no template variable, no suite.prod.json.
- One file runs against a per-PR deployment of your bot, then against the release artifact at deploy time.
- One file runs over voice, then over chat, if your agent does both.
The moment environment details leak into the test file, you have N files drifting against each other, and you've reinvented the original problem in a nicer font.
How do you automate voice agent regression testing with a coding agent?
A test suite in a repo is only better than a dashboard if it stays current. Otherwise you've just moved the drift.
So don't maintain it by hand. Point a coding skill at it: a repo-local skill that runs on merge, reads the diff, and updates the affected cases in the same pull request that changed the agent.
The important part isn't automation, it's reviewability: the prompt diff and the eval diff land in the same PR, and one human reads both.
The skill needs rules, because an agent editing its own tests has an obvious degenerate solution: weaken the assertion until it goes green. Nobody has to teach it that; it will find it. The ones that matter:
Never weaken a red. The skill may not relax an expected_outcome to make a failing case pass. If a case is genuinely wrong, that's a human decision made out loud, with a reason.
Don't touch what the diff didn't touch. No opportunistic edits to unrelated cases.
Prefer extending a case over adding one. A suite you gate every PR on has to stay small. Six parallel cases is a gate. Sixty is a nightly job nobody reads.
Only assert what the judge can see. For voice, the judge reads a transcript. Don't assert that ambience played or that a code was spelled out: speech-to-text will happily normalize “7 3 9 1” into “7391” and fail you on a call that went perfectly.
Buy determinism where it's cheap. For scripted flows, fix both sides' messages. Then a pass is guaranteed when the agent is right, and a failure means the agent, not the dice.
Every edit the skill proposes is validated before it lands (see the next section), so a malformed case or a metric slug that no longer exists never reaches a human's eyeballs.
How do GitHub Actions fit into voice agent testing?
With the suite as a file, the pipeline splits cleanly into a free tier and a paid one.
Every commit: validate. dry_run=true checks the whole spec, resolves every reference against the target agent, prices the run, and creates nothing. No calls, no credit. It catches a malformed file, a deleted personality, a metric that no longer exists, in about a second, for nothing.
Branches that matter: run it for real. Real calls, real judges, a pass/fail your merge gate can depend on. Cekura's GitHub Actions guide has a workflow you can copy as-is, and GitHub's own docs on what triggers a workflow are worth reading alongside it when you're deciding which branches qualify.
Here's the difference between wiring a generic GitHub Actions setup to voice agent testing and treating the suite as code the way Cekura does:
| Dimension | Generic GitHub Actions setup | Cekura tests-as-code |
|---|---|---|
| Where the suite lives | Scenario IDs listed in the workflow YAML | Versioned JSON file committed next to the agent |
| Cost to validate a change | Runs the full suite, or nothing | dry_run=true prices the run for free; no calls placed |
| Who updates the tests | A person edits the workflow or a dashboard | A coding skill proposes the edit in the same PR as the prompt diff |
| What a malformed case does | Fails at runtime, mid-suite | Rejected before anything runs, error keyed to scenarios[3].metrics[1] |
| Failure vs. infrastructure timeout | Both usually just fail the job | Two distinct terminal states: fail-and-block vs. retry-once-then-neutral |
Two details that make this pleasant rather than painful:
Errors come back all at once, keyed to a location in the file, e.g. scenarios[3].metrics[1], instead of one at a time. A validator that stops at the first mistake costs one pipeline round trip per mistake.
Metrics that would evaluate nothing are a hard error, not a warning. A metric can be valid, in the right project, referenced correctly, and still evaluate nothing if it isn't enabled for the agent you're testing. In a dashboard you'd eventually notice the blank column. In CI, that's a green build that tested nothing, so the build fails instead.
What does CI/CD testing for voice AI agents actually run on every commit?
Two traps worth flagging, neither specific to any vendor:
Skipped isn't passed. If you make the eval a required check and skip the job when nothing relevant changed, the check sits pending forever and nothing merges. Always reach a terminal state: run, exit 0, and print why the suite wasn't required.
A failure and a wobble aren't the same red. A judge scoring a call red should fail the build, every time, no retry. A run that never completed (agent didn't connect, provider timed out) should retry once and then exit neutral with a loud comment. An infra wobble must not read as a code failure, and it must not silently pass either.
That distinction isn't theoretical. Per Cekura's benchmarks, infrastructure-clean call rates across one tested cohort ranged from 100% (ElevenLabs) down to 82.93% (Vapi): real evidence that a meaningful share of CI runs will fail for reasons that have nothing to do with the code, on any platform.
What was missing from the voice agent self-improvement loop?
Tests as code closes the last gap in the self-improvement loop most teams are already building toward:
- Production failure
- Cluster into a reproducible failure mode
- Generate a scenario that reproduces it
- Fix the prompt
- Verify the fix
- Regression-test forever
Most of that has been buildable for a while. You can mine call logs, cluster failures, generate scenarios, run them, and have a coding agent propose the prompt fix.
The missing link was the last one. An agent could change the prompt, but it couldn't meaningfully change the tests, because the tests weren't files. Every closed loop ended with a human going to a dashboard and clicking. Cekura's own self-improving voice agents work covers the diagnose-and-fix half of this loop; this piece is about the half that makes the fix stick.
Once the suite is code, the loop closes inside your normal workflow:
- A production failure becomes a scenario: a new case in the spec file.
- The fix and the case that proves it land in the same pull request.
- CI validates the file for free, then runs it for real.
- The case stays in the suite, so the same failure can never ship twice.
And the human is still in it, in exactly the right place: reading a diff. That's the boundary worth keeping. An agent that writes the code and grades the code will eventually find that the cheapest path to green is a weaker test. Three things stand in the way: a validator it can't sweet-talk, a hard rule against relaxing a failing assertion, and a person reading the PR. Automate the first two. Keep the third.
How do you start with tests as code?
If you already have evaluators, this is a fifteen-minute change to your AI agent testing setup:
- Export your suite to JSON and commit it next to your agent.
- Add a dry_run=true call to your existing PR workflow. It costs nothing and never places a call.
- Point your live-call job at the file instead of at a list of scenario IDs.
- Add the update skill so the suite maintains itself on merge.
Step 3 is the one that matters most. Once the suite is a file, changing your agent and changing its tests is one commit, one review, one revert.
Frequently asked questions
How do you automate voice agent regression testing with a coding agent?
Point a repo-local coding skill at the test-suite file so it runs on merge, reads the diff, and updates only the cases the diff touched, under a rule that blocks it from weakening a failing assertion. Every edit it proposes is validated before it lands in the same pull request as the prompt change.
What does CI/CD testing for voice AI agents actually run on every commit?
A free dry_run validation pass that resolves every reference in the suite and prices the run without placing a single call. The real calls, with real judges, run only on the branches that matter for your merge gate.
Can a coding agent update its own tests safely?
Only within limits. It can extend or add cases the diff touched, but a hard rule blocks it from relaxing an expected_outcome to make a failing case pass, and every edit is validated before a human reviews it.
What's the difference between a failed voice agent test and an infrastructure timeout?
A judge scoring a call as wrong should fail the build every time, with no retry. A call that never completed, such as a provider timeout, should retry once and then exit neutral with a loud comment, since an infra wobble must not read as a code failure and must not silently pass either.
Where should an AI agent's test suite live?
As a versioned JSON file in the same repo as the prompt and tool definitions it tests, not behind a dashboard login, so a pull request that changes the prompt and its test suite carries both in one diff.
Cekura tests, monitors, and helps voice and chat agents self-improve.
If you want to see AI agent testing like this running against your own agent, start a free Cekura trial: the dry-run check costs nothing to try.
Reference: Spec schema (v1) · GitHub Actions setup