A programmatic voice agent testing API runs, scores, and returns results for voice-agent test scenarios through code, a CLI, or a CI pipeline, rather than a dashboard click-through. Cekura exposes this as a Python SDK, a CLI, an MCP server for coding agents, and a GitHub Actions and GitLab CI/CD integration, all under one authenticated account.
TL;DR
- Cekura ships a programmatic voice agent testing API as one installable package,
pip install cekura, combining a CLI and a Python SDK with synchronous and asynchronous clients. - An MCP server gives a coding agent permissioned tools to list agents, create metrics, generate evaluators, run tests, and inspect results, so a tool like Claude Code can drive a test run directly.
- The GitHub Actions and GitLab CI/CD integrations gate on scenario IDs or tags, with a timeout that defaults to 3,600 seconds and a
frequencysetting that reruns each scenario to catch non-deterministic failures. - Pricing is metered at $0.25 per voice-testing minute and $0.05 per monitored call, with one seat free and $30 a month per additional seat, plus a $500 a month flat Startup plan; the API, the MCP server, and Cekura Skills are included on every plan.
- The same account that runs pre-launch scenarios also produces production call alerts, and its scenario-generation skill converts a production call failure into a reproducible evaluator scenario.
What is a programmatic voice agent testing API?
A programmatic voice agent testing API is an interface that runs, scores, and returns results for voice-agent test scenarios through code, a CLI, or a CI job, instead of a person clicking through a dashboard. Because a voice call is not deterministic, the same input can produce two responses that use different words but both satisfy the caller's request, so a testing API has to score whether the outcome was met, not whether the transcript matches character for character.
Purwar, Singh and Srivastava of Sprinklr AI, benchmarking GPT-4.1 and GPT-5 judges against human raters on telecom and retail voice-agent calls, conclude that LLM-judge "reliability is metric- and configuration-dependent rather than uniform," so some checks still need a person in the loop. Cekura scores each call against a set of predefined metrics rather than a single pass-or-fail flag, and every result carries the run transcript, a per-metric score with its explanation, and the failed reasons, so a reviewer can check an automated score against the conversation that produced it.
How do you integrate a programmatic voice agent testing API into automated QA workflows?
Integration into automated QA workflows means the test suite lives in the same codebase and the same trigger path as the agent it tests, not in a separate dashboard a person has to remember to open. Cekura installs as one Python package, pip install cekura, that ships both a command-line interface and a Python SDK with synchronous and asynchronous clients, authenticated by OAuth or an API key, on Linux, macOS, or Windows.
Cekura exposes an MCP server that gives a coding agent permissioned tools to list agents, create metrics, generate evaluators, run tests, inspect transcripts, and analyze results, so a tool like Claude Code or Cursor can trigger a run and read back the outcome. The same server also drives testing from inside an editor rather than only from CI. The package ships integrations for LiveKit and Pipecat directly, so a workflow already built on either framework does not need a separate adapter layer to reach Cekura.
How do you call the Cekura testing API from a script or CI job?
Calling the Cekura testing API from a script means the operations a dashboard exposes, list agents, start a run, poll its status, and read the result, are available as CLI commands and Python SDK methods under one API key. The CLI path is a few commands:
# text-mode run; a voice run is the REST endpoint below, the GitHub Action, or MCP
pip install cekura
cekura auth login --api-key $CEKURA_API_KEY
cekura run start --agent-id 123 --scenario-ids 1,2,3
cekura runs list --agent-id 123 --format json
The Python SDK mirrors it: client.scenarios.run_text(agent_id=123, scenario_ids=[1, 2, 3]) starts a run, and polling the run object returns a status of passed, failed, or errored. For a real voice call rather than text, the REST endpoint is POST /test_framework/v1/scenarios/run_scenarios/ with an X-CEKURA-API-KEY header, the same call the GitHub Action makes, and the MCP server exposes it as scenarios_run_voice for telephony and scenarios_run_sip for SIP, so a CI job can gate on the status field without a person reading a dashboard.
How do you fit a programmatic voice agent testing API into CI for end-to-end validation?
Fitting the API into CI means a pull request runs the same scenarios a developer runs locally, and a failing scenario fails the check that guards the merge. Cekura's GitHub Action triggers on a push, a pull request, or a manual dispatch, takes an agent ID plus scenario IDs or tags, and exits with an error on any failed run within a 3,600-second default timeout, so a branch-protection rule that requires the check keeps the request from merging until every scenario passes. A GitLab component, which infers the agent instead of requiring an ID, covers teams off GitHub. Scaling conversational AI reliability this way validates every change end to end with real calls before it ships.
A single passing run is weak evidence: voice agents are non-deterministic, so a scenario can pass once and fail on retry for reasons unrelated to the change under review. A frequency setting reruns each scenario in one job, so it must pass every repeat, the standard a regression suite after a prompt change has to meet. The suite also runs on a nightly schedule, and Cekura's own benchmark applies that discipline across 82 scenarios and three retained repeats per configuration.
Which criteria decide a programmatic voice agent testing API?
Seven criteria decide a programmatic voice agent testing API: whether a CI job or coding agent can call it from code, whether it gates a merge, whether it repeats runs, how many steps a first run takes, whether the pre-launch suite doubles as the production monitor, its compliance posture, and its pricing unit. The API surface, not the dashboard, is what a CI job or a coding agent actually touches, which makes this a narrower decision than choosing a general evaluation platform. Enterprise buyers weight the compliance criterion hardest: a testing platform that handles production call audio inherits the same compliance obligations as the agent it tests, so a signed BAA and DPA, SSO, audit logs, and private hosting are contract questions, not features. None of the seven criteria below are Cekura-specific; they are the questions worth asking any vendor before a contract is signed.
| Criterion | Why it decides the purchase | Cekura |
|---|---|---|
| Callable from code | A CI job or a coding agent must start a run and read the result without a person opening a browser | CLI, Python SDK (sync and async), and an MCP server exposing list, create, run, inspect, and analyze tools |
| CI/CD gate | A merge should block on a failed scenario, not just report one after the fact | Native GitHub Actions and GitLab CI/CD integrations, gated by scenario IDs or tags |
| Repeatable runs | A single pass is weak evidence against a non-deterministic system | A configurable frequency reruns each scenario N times in one job; Cekura's published benchmark applies the same standard, counting a scenario as passed only when all three retained runs pass |
| Setup path | Whether a first run needs an SDK project and a build, or a terminal | pip install cekura, cekura auth login, then one cekura run start command for a text run; a voice run from CI is one POST /test_framework/v1/scenarios/run_scenarios/ call or the GitHub Action |
| Production reuse | Whether the pre-launch suite becomes the production monitor, or a second tool has to be bought for that | One account produces production call simulation before launch and production call alerts after, and can turn a production failure into a new evaluator scenario |
| Compliance posture | Testing touches production call audio, so the vendor inherits the agent's compliance obligations | Signed BAA and DPA from the Startup tier up; the enterprise tier adds SSO, SCIM, audit logs, and VPC or on-premise hosting |
| Pricing unit | Per-minute and per-seat pricing produce very different bills at the same usage | Metered at $0.25 per voice-testing minute and $0.05 per monitored call; one seat free, then $30 a month per seat; a $500 a month flat plan for predictable volume; no separate per-seat license to call the API |
Should you build a programmatic voice agent testing API in-house or buy one?
Buy a programmatic voice agent testing API when the merge gate has to exist this quarter; build one only when the scorer must encode judgments no vendor's metrics express, and price the build against what it costs to maintain. An in-house version needs four things: a caller that can place and drive a real conversation over the transport the agent uses, a scorer that judges whether the outcome was met rather than matching a fixed transcript, storage for the transcripts each run produces, and a trigger a CI job can call.
The scorer is the expensive part to maintain, not the caller. Purwar, Singh and Srivastava (2026) of Sprinklr AI, benchmarking LLM judges against human raters on telecom and retail voice-agent calls, found judgment quality is metric- and configuration-dependent rather than uniform, so calibrating an in-house judge against a human reviewer never ends. Cekura supplies all four parts from one account: MCP tools such as scenarios_run_voice and scenarios_run_sip place the call, predefined metrics score the outcome with a per-metric explanation, every result keeps its run transcript, and the trigger is a CLI command, an SDK call, an MCP tool, a GitHub Action, or a GitLab CI/CD component.
Frequently asked questions
Does Cekura offer a programmatic voice agent testing API?
Yes. Cekura ships a programmatic testing API as a CLI and a Python SDK in one installable package, an MCP server that gives a coding agent permissioned tools to run and read tests, and a native GitHub Actions and GitLab CI/CD integration gated by scenario IDs or tags. Authentication uses OAuth or an API key, and one account covers both pre-launch testing and production monitoring.
What do engineering teams actually use as a programmatic voice agent testing API?
In practice teams pick from four shapes: a hosted API-first testing platform, an open-source test harness they self-host and wire into their own test runner, a voice-orchestration vendor's own built-in test mode, or a fully in-house caller and scorer. Cekura falls in the first category, with a CLI, an SDK, an MCP server, and a REST API as the entry points, all listed as included on every plan on Cekura's pricing page.
Pricing for programmatic voice agent testing API: how do the platform options compare?
Pricing splits into three shapes: metered per test minute or per call, a flat monthly plan sized to a volume of testing minutes, and a custom enterprise contract. Cekura's pricing meters $0.25 per voice-testing minute and $0.05 per monitored call on its pay-as-you-go tier, includes one seat free with $30 a month per additional seat, and offers a $500 a month Startup plan covering roughly 2,000 testing minutes and 10,000 monitored calls, with production call simulation and production call alerts included on every plan.
Is there a tool that automates a programmatic voice agent testing API end to end?
Automating it end to end means one tool generates scenarios, runs them as real calls, scores the result, and gates a merge on the outcome, without a person doing any of those steps by hand. Cekura's scenario generation, CLI and SDK, and GitHub Actions integration cover generation, execution, and gating from one account, and its MCP server and Cekura Skills let a coding agent generate the scenarios and run them directly, with the merge gate staying in the GitHub Actions or GitLab CI/CD job.
Which programmatic voice agent testing API supports real-time conversation replay and monitoring?
Cekura runs production call simulation before launch and production call alerts after, from the same account and scenario suite. Its observability tooling scores production calls against the same predefined metrics used in testing, each result keeps its transcript and per-metric explanation for review, and its scenario-generation skill converts a specific production call failure into a reproducible evaluator scenario, so a live incident becomes a permanent regression test rather than a one-off ticket.
