An SDK for automating voice agent tests is a client library that defines test scenarios in code, triggers runs against a live agent, and reads back structured results instead of a pass or fail. Cekura exposes that surface three ways: a Python SDK and CLI, a REST API, and a GitHub Action.
TL;DR
- The framework SDK you already have tests text, not audio. LiveKit's test framework runs in text mode under pytest or Vitest, and Pipecat Evals skips the audio services by design.
- A platform test suite runs the audio path instead. Vapi connects a testing agent to your voice agent on a real phone call, then grades the transcript with a language model.
- Automating the text path and automating the audio path are different jobs, and a serious suite ends up doing both at different points in the loop.
- Cekura triggers and reads runs over REST at api.cekura.ai, and its GitHub Action is gated by an agent ID plus scenario IDs or tags, with a 3600 second default timeout.
- A single passing run is not a result. Cekura's benchmark runs each scenario three times and reports the share of scenarios that passed all three.
What does an SDK for automating voice agent tests actually automate?
An SDK for automating voice agent tests automates three things: defining a scenario in code, triggering a run, and reading structured results back into your own tooling. Everything else is a variation on those three.
The first is the part teams underestimate. A scenario is not a prompt. It is a caller situation with a goal, a personality, mock tool responses, and the criteria the transcript gets graded against. Cekura documents that construction in The Complete Cekura Scenario Testing Guide.
The second decides where tests can run at all. A synchronous trigger blocks until the call finishes, which suits a local script. An asynchronous trigger returns a run identifier immediately, which is what a CI job or a scheduler needs to avoid holding a runner open for the length of a phone call.
The third decides what you can do with a failure. A boolean tells you a test broke. A structured result tells you which metric moved, on which turn, with the transcript attached. Cekura documents its metric set in A Developer's Guide to Voice AI Evaluation Metrics.
Which programmatic test interface should you reach for?
Reach for the framework's own test SDK during local iteration, and for a harness that places real calls before anything ships. The two interfaces answer different questions, and the table below sets out what each one actually runs.
| Platform | Programmatic test surface | What a run exercises | How you invoke it |
|---|---|---|---|
| LiveKit Agents | Built-in agent test framework | Text mode: specific messages, tool calls, arguments and handoffs, asserted turn by turn | pytest (Python) or Vitest (Node.js), locally or in CI |
| Pipecat | Pipecat Evals | Text mode, exercising the pipeline and context handling while skipping the audio services | pipecat eval run on a YAML file, or as a library |
| Vapi | Voice test suites | A real phone call between a testing agent and your voice agent, recorded and transcribed | A test suite whose transcript is graded against your rubric by a language model |
| Cekura | Python SDK and CLI, REST API, GitHub Action, MCP server | Real calls over telephony, SIP, or a provider's WebRTC transport | A script, a CI job, a scheduler, or a coding agent |
The split in the third column is the one that matters. Text-mode tests are cheap and deterministic, so they belong on every commit. Audio-path tests cost a real call and real seconds, so they belong on a pull request gate, a nightly schedule, and a release candidate. Choosing one interface for both jobs is where suites go wrong: a text-only suite ships an agent that mishandles a barge-in, and an audio-only suite is too slow for anyone to run while writing code.
Two questions settle which column you need. Does the change touch prompt logic and tool wiring, or does it touch the voice stack, the transport, or turn-taking? And does the result need to gate a merge, which means the interface must be callable from a CI runner with no person present? A tool definition change is answered in text mode in seconds. A model swap, a new voice, or an endpointing setting is not answerable there at all.
Do the voice framework SDKs already test your agent?
The voice framework SDKs test the agent's logic, not its audio, and both major open-source frameworks say so in their own documentation.
LiveKit's agent testing framework asserts on "Specific messages, tool calls, arguments, and handoffs that you assert on, turn by turn", and "Runs locally or in CI with pytest or Vitest", producing "Text-based tests with deterministic results". LiveKit's own framing of the tradeoff is explicit: "Text mode is the most cost-effective and deterministic way to test agent behavior."
Pipecat Evals takes the same position. Pipecat describes the product as "Behavioral testing for your agents: scripted conversations, semantic assertions, and an LLM judge", and states that "Text mode exercises your agent's actual pipeline and context handling while skipping the audio services". Assertions are semantic rather than literal, since "a judge LLM evaluates the agent's response against a natural-language criterion". Pipecat runs its own suite at scale: "Before every release, an eval suite drives 100+ example agents end to end."
Both are genuinely useful, and neither is measuring endpointing, barge-in latency, or what a codec does to a spoken account number.
What does a platform test suite run that a framework SDK does not?
A platform test suite runs the call itself, which is the layer a text-mode SDK cannot reach.
Vapi's voice testing documentation describes the mechanism plainly: "Our platform connects two AI agents - your voice agent and our testing agent - on a real phone call, following your predefined scripts to evaluate performance under various scenarios." The grading happens after the fact, because "The entire call is recorded and transcribed for evaluation" and then "the transcript is evaluated against your rubric by a language model (LLM)".
That design buys the failure modes text mode misses. A real call carries turn-taking, so an agent that talks over the caller fails it. It carries transport, so a dropped leg or a silent stretch shows up. It carries speech recognition on synthesised audio, so a digit misheard through a codec surfaces as a wrong tool argument rather than a clean transcript.
The cost is time and money per run, which is why this layer sits behind a gate rather than in a watch loop.
What does Cekura expose programmatically?
Cekura exposes the same test suite through four entry points, so the interface changes with the trigger while the scenarios stay fixed.
Cekura serves a REST API at api.cekura.ai, authenticated with an X-CEKURA-API-KEY header. Runs come back through a bulk runs endpoint, GET /test_framework/v2/runs/bulk/, which takes a comma-separated run_ids parameter and returns run objects carrying transcripts, evaluation results, status and metadata.
Cekura ships its CLI and Python SDK as one installable package with sync and async clients, documented on programmatic voice agent testing API.
Cekura also publishes a GitHub Actions workflow rather than a generic webhook, triggering on a push to main, on pull requests, or on manual dispatch, and gating on scenario IDs or tags instead of running the whole suite every time. Cekura calls it "a simple, dependency-free solution that uses standard tools available on GitHub runners".
Cekura also exposes an MCP server, the entry point a coding agent drives, covered in how to automate voice agent regression testing with a coding agent.
Why is a single passing run not enough to trust an automated suite?
A single passing run does not establish that a scenario passes, because voice agents are non-deterministic and an SDK that returns one boolean per scenario hides that.
Cekura's own benchmark methodology is built around the problem. Each scenario ran three times, and the headline metric is pass cubed, which Cekura defines as "the share of 82 scenarios where all three retained runs passed". Cekura also refuses the usual escape hatch: "Failures remain in the results. Calls that did not connect or produced no transcript stay in the denominator."
Applied to your own suite, that method has two consequences. Repeat count belongs in the scenario definition, not in a retry wrapper, because a retry that masks an intermittent failure deletes the signal you were testing for. And infrastructure failures belong in the denominator, since a call that never connected is a product failure to the person who dialled.
Cekura sets out how these layers stack in Engineering Reliability: Why Your Voice AI Needs a CI/CD Pipeline.
Frequently asked questions
Can you run voice agent tests from CI?
Yes. Cekura publishes a GitHub Action, cekura-ai/cekura-github-actions@v1.0.0, that triggers on a push to main, on pull requests, or on manual dispatch. It takes an agent ID and an API key, selects scenarios by ID or by tag, and defaults to a 3600 second timeout. LiveKit's text-mode framework runs in CI too, under pytest or Vitest.
What does an SDK for automating voice agent tests need to expose at minimum?
Three things: a way to define a scenario in code, a way to trigger a run synchronously or asynchronously, and a way to read back structured results. Synchronous triggers suit local scripts. Asynchronous triggers suit CI jobs and schedulers, which cannot hold a runner open for the length of a phone call.
Do LiveKit and Pipecat's own test frameworks exercise audio?
No, both run in text mode. LiveKit states that "Text mode is the most cost-effective and deterministic way to test agent behavior", and Pipecat states that "Text mode exercises your agent's actual pipeline and context handling while skipping the audio services". Both are sound choices for logic and tool-call coverage, and neither measures endpointing, barge-in handling, or codec effects.
How do you trigger a Cekura test run from your own code?
Through the REST API at api.cekura.ai, authenticated with an X-CEKURA-API-KEY header, or through the Python SDK and CLI that Cekura ships as one installable package with sync and async clients. Results come back as run objects carrying transcripts, evaluation results, status and metadata, so a failure is diagnosable without opening the dashboard.
Why should each scenario run more than once?
Because one run does not separate a real regression from ordinary variance. Cekura's benchmark runs each scenario three times and scores on pass cubed, "the share of 82 scenarios where all three retained runs passed". Put the repeat count in the scenario definition rather than in a retry wrapper, or an intermittent failure gets masked instead of measured.
