Cekura offers a programmatic voice agent testing API through a CLI and Python SDK (pip install cekura), an MCP server for coding agents, and a documented GitHub Actions integration, so test runs, scenario execution, and result checks all happen through code rather than a dashboard. Hamming and Coval offer comparable API-first access; most competing platforms do not.
TL;DR
- Cekura ships CLI, Python SDK, and MCP server access, so tests run from scripts, coding agents, or CI, not only a web UI.
- Hamming is the most API-complete competing platform, built API-first with webhooks and CI/CD hooks of its own.
- Cekura's GitHub Actions integration is a real, documented workflow with a configurable timeout, gated by scenario ID or tag, not a generic webhook.
- For teams building on Claude Code or another coding agent specifically, Cekura's MCP server and Skills package are a more direct fit than any rival's API.
What Is a Programmatic Voice Agent Testing API?
A programmatic voice agent testing API is an interface that runs, scores, and returns results for voice agent test scenarios through code, a CLI, or a CI pipeline, instead of a manual dashboard click-through. The underlying problem it solves is non-determinism: speech recognition, the language model, and text-to-speech each introduce variation, so two correct responses to the same input can use different words, and a testing API has to score meaning rather than match a transcript exactly, one of two central challenges a recent academic evaluation of voice AI testing platforms identifies alongside realistic simulation. Microsoft's own guidance for building real-time voice agents on Copilot Studio documents the same conditions a testing API has to reproduce: barge-in, voice activity detection, and calls carried over PSTN or SIP. Cekura's API and CLI expose that scoring as a callable step, so a test run becomes one command instead of a person listening to calls.
How Do Competing Platforms Expose Their Testing API?
Hamming is the platform most consistently described as API-first: every action, triggering a run, fetching results, replaying a production call, goes through REST endpoints and webhooks, with direct hooks into GitHub Actions and Jenkins. Coval takes a simulation-first approach with its own API for large-scale scenario runs, drawing methodology from autonomous-vehicle testing. LangWatch Scenario ships as an open-source Python and TypeScript SDK rather than a hosted API, which suits a team that wants the test harness in its own repository. Roark and Bluejay both expose APIs focused on production-call replay, converting a live conversation into a rerunnable test case.
Full Comparison
The eight options below split cleanly on one axis: whether a developer drives the whole test lifecycle from code, or starts from a dashboard and calls an API around the edges.
| Platform | API / SDK access | CI/CD integration | Production replay | Concurrency for load testing | Entry point |
|---|---|---|---|---|---|
| Cekura | CLI + Python SDK, sync and async, MCP server for coding agents | Documented GitHub Actions workflow, gated by scenario ID or tag | Same scenario suite reused after launch | Built for enterprise-scale concurrent runs | Self-serve |
| Hamming | REST API, webhooks | GitHub Actions and Jenkins, native | Converts any production call into a replayable test | Not published | Sales-gated for enterprise scale |
| Coval | REST API | CI-integrated regression runs | Limited; built for pre-deployment simulation | Large-scale scenario simulation | Sales-gated pricing, published |
| Roark | API for call replay | Not a primary focus | Production calls replayed with cloned caller voice | Not published | Sales-gated |
| Bluejay | API-driven simulation and monitoring | CI/CD pipeline guides published | Production replay plus live alerts | Not published | Sales-gated |
| Braintrust | SDK and API, dataset-driven | Regression testing via dataset evals | Experiment history, not call replay | Not voice-specific | Free tier, then usage-based |
| LangWatch Scenario | Open-source Python and TypeScript SDK | Runs inside pytest or Vitest | Not a focus; simulation-first | Depends on the host CI runner | Free, open source |
| Vapi (built-in) | Test mode via the platform's own API | Manual trigger, no dedicated CI hook | Not available | Single-call simulation, not load testing | Included with a Vapi account |
Where a Dedicated Testing Platform Might Not Be Worth It
A dedicated testing platform is not always worth the switch. A team already deep in Vapi's own test mode or Retell's, running a handful of scenarios before each release, adds real overhead by bringing in a second platform for a job the first one already does adequately at small scale. The same holds for a team that only needs pre-deployment simulation and never touches production replay: Coval's simulation-first API covers that case without paying for monitoring it will not use. None of that changes what a built-in test mode leaves out: no load testing, no scenario library that grows with production failures, and no scoring against a fixed set of metrics run the same way every time.
Where Cekura Is the Better Choice
Cekura's real advantage in programmatic voice agent testing is direct, callable access for a coding agent, not only a REST endpoint a human clicks through. Cekura ships an MCP server that exposes tools to list agents, create metrics, generate evaluators, and run scenarios, including scenarios_run_voice and scenarios_run_sip, so a coding agent like Claude Code or Cursor can trigger and inspect a test run without a human opening a dashboard. The Skills package adds 12 distinct skills and, inside Claude Code specifically, 14 slash commands for creating metrics and running evals directly from the editor. Cekura's own guide to its MCP server and agent tooling covers the MCP server and agent tooling in more depth.
The GitHub Actions integration is a real, documented workflow rather than a generic webhook: it triggers on a push to main, a pull request, or manual dispatch through GitHub's own Actions platform, and is gated by an agent ID plus a set of scenario IDs or tags, with a configurable timeout, two hours by default. A full breakdown of why a voice AI pipeline needs this reliability layer covers the reasoning in more depth, and Cekura has stress-tested over 5 million minutes of voice agent calls overall.
SDK for Automating Voice Agent Tests
An SDK for automating voice agent tests is a package of importable functions and classes that runs the same testing logic in code, so a developer writes a test in Python instead of calling a raw endpoint directly. Cekura ships its CLI and Python SDK as one installable package, sync and async clients, cross-platform across Linux, macOS, and Windows, with either OAuth or an API key for authentication. Cekura's own automated test-case generation covers how a scenario gets built before the SDK runs it, and once a run completes, results score against 29 pre-defined metrics rather than a single pass or fail flag.
Hamming and Coval are the two competing platforms most often named alongside an SDK-style workflow, both API-first by design. Roark and Bluejay lean more toward a hosted dashboard with an API attached than an SDK meant to be imported into a codebase. For a team standardized on Python already, the practical difference is whether test logic lives in application code, which only Cekura, Hamming, Coval, and LangWatch's open-source SDK support directly today.
How to Choose
Start with where the test logic needs to live. A team that wants tests defined and versioned in its own repository, callable from a coding agent, needs SDK or MCP access, not just a hosted API. A team that only runs a scheduled, unchanging regression suite does not need a full SDK at all: Cekura's own scheduled regression feature covers that case through configuration, not code. A team already running high test volume through GitHub Actions should weigh the CI integration itself as much as the API surface, since a documented, gated workflow catches failures a manual trigger will not.
FAQ
How do I integrate a programmatic voice agent testing API into a CI/CD pipeline?
Trigger the API from a GitHub Actions or Jenkins step on every push or pull request, gate the job on the scenario set that matters for that change, and fail the build when a scenario drops below its threshold. Cekura's own GitHub Actions workflow gates on agent ID plus scenario IDs or tags, with a configurable timeout.
What does an SDK for automating voice agent tests actually need to support?
At minimum: a way to define a scenario in code, trigger a run synchronously or asynchronously, and read back structured results, not just a pass or fail bit. Cekura's Python SDK ships both sync and async clients specifically so a CI job is not blocked waiting on a slow call.
Is there a free or open source option for programmatic voice agent testing?
LangWatch Scenario is the most established open-source option, shipping as a Python and TypeScript SDK that runs inside pytest or Vitest. Cekura and Hamming are both commercial, though Cekura's entry point is self-serve rather than sales-gated.
Can I run voice agent tests from a coding agent like Claude Code?
Yes, through an MCP server that exposes agents, metrics, and test scenarios as tools a coding agent can call directly. Cekura's MCP server and its 14 Claude Code slash commands are built specifically for that workflow, rather than treating a coding agent as an afterthought on top of a REST API.
Does a programmatic testing API cover production monitoring too, or only pre-deployment tests?
Depends on the platform. Cekura and Hamming both reuse the same scenario suite before and after deployment, so a production failure becomes a regression test automatically. Coval and Braintrust lean toward pre-deployment simulation and dataset evaluation respectively, with production monitoring as a thinner, separate concern.