Free and open source voice agent testing tools include Pipecat's eval harness, the LiveKit Agents test framework, ServiceNow's EVA-Bench, voicetest and promptfoo. Most score text, or only agents built on one framework. Cekura tests the full audio path over phone, SIP and WebRTC, starts with 300 free credits, and publishes seven judge prompts under Apache-2.0.
TL;DR
- Cekura starts free with 300 credits, which its pricing page equates to about 60 minutes of voice testing, with no card and one free seat, then bills $0.25 per voice testing minute (rates read 23 September 2026).
- Pipecat's release-eval harness is the most fully free option: its default judge, caller voice and transcriber run locally with no keys and no per-run cost, but it drives Pipecat bots only.
- The LiveKit Agents test framework runs text-based, deterministic tests in pytest or Vitest, and its audio simulations, in beta for Python agents, run on LiveKit Cloud.
- A free licence is not a free run: EVA-Bench's authors call a full run a non-trivial compute investment, because the simulated caller, the agent and the judge all bill.
- One run is not a result: in Cekura's benchmark, the LiveKit and Pipecat configurations, each chosen by its provider, passed all three retained runs on 70.73% and 63.41% of 82 scenarios, with calls that did not connect counted as failures.
Which open source voice agent testing tools exist, and what does each one test?
Open source voice agent testing tools fall into three groups: framework harnesses that test agents built on one framework, simulators that play a caller against any agent, and evaluation libraries that score text. Pipecat's release-eval harness and the LiveKit Agents test framework form the framework group. ServiceNow's EVA-Bench and voicetest simulate callers: EVA-Bench runs bot-to-bot audio conversations, while voicetest ships with audio evaluation switched off by default. Voice Lab states that it supports only the text part of a voice agent, and its repository has not been updated since June 2025. promptfoo and DeepEval score prompts and model outputs. τ2-bench evaluates realtime model providers on its own customer-service domains, which benchmarks a model rather than your deployed agent. Cekura sits outside the open source column as a commercial platform, but it publishes seven transcript-judge prompts under Apache-2.0 and gives new accounts free credits, so the table lists it on the terms that apply. Anyone building on Deepgram has a narrower question, and our Deepgram voice agent testing page covers how to run and score scenario calls against it.
| Tool | Licence | What it exercises | Where it runs | What you still supply |
|---|---|---|---|---|
| Pipecat release evals | BSD-2-Clause | Scripted and simulated conversations, in text or synthesized audio | Locally; default judge, caller voice and transcriber need no keys | A Pipecat bot and that bot's own service keys |
| LiveKit Agents test framework | Apache-2.0 | Messages, tool calls, arguments and handoffs, turn by turn, in text | pytest or Vitest, locally or in CI; simulations over text or audio run on LiveKit Cloud (beta, Python agents only) | An LLM for the agent and the judge |
| EVA-Bench (ServiceNow) | MIT | Bot-to-bot audio conversations, 213 scenarios, noise, accent and connection perturbations | Locally, calling hosted models | Simulated caller (ElevenLabs or OpenAI Realtime), speech and judge API keys |
| voicetest | Apache-2.0 | LLM-simulated users against imported Retell, Vapi, LiveKit, Bland or Telnyx agents; audio evaluation off by default | CLI, web UI or CI | Any LiteLLM-compatible model |
| Voice Lab | Apache-2.0 | The language model and prompt only | Locally | An OpenAI key; no updates since June 2025 |
| τ2-bench (voice mode) | MIT | Realtime providers (OpenAI, Gemini, xAI) on fixed customer-service domains | Locally | Provider keys; it benchmarks models, not your agent |
| promptfoo, DeepEval | MIT, Apache-2.0 | Prompts and model outputs as text | Locally or in CI | Transcripts captured somewhere else |
| cekura-ai/metrics | Apache-2.0 | Seven LLM-judge prompts over a transcript | Inside your own code | The model call and the transcript |
| Cekura platform | Commercial, free start | Simulated calls over phone numbers, SIP and WebRTC, scored by LLM-judge and Python metrics | Hosted | Nothing for the first 300 credits, then $0.25 per voice testing minute |
Why do free open source voice agent testing tools still cost money to run?
A free licence covers the code, not the calls the code makes. Every voice test has three billable parts: a simulated caller, the agent under test, and a judge that scores the result. The EVA-Bench paper from ServiceNow AI Research itemises all three: its default caller is ElevenLabs' hosted ElevenAgents, proprietary agents bill per token and per second of audio, and four of its main metrics use an LLM judge. One full run is 1,875 evaluation samples per model, and first author Tara Bogavelli and 12 co-authors conclude that researchers "should therefore treat full EVA-Bench runs as non-trivial compute investments." Pipecat's harness is the exception: its defaults run locally with no keys and no per-run cost, though each bot still needs its own speech and model keys. Cekura bills the simulated caller at $0.25 per voice testing minute, LLM-judge metrics at 0.2 credits per evaluation, and Python metrics at nothing, while your agent's own provider bills as usual.
How many runs does a voice agent test need before the result means anything?
A voice agent test result is a rate, not a verdict, because the same scenario can pass once and fail on the next attempt. EVA-Bench measured this across 12 systems over five trials per scenario: the median gap between peak performance (pass@k) and reliable performance (pass^k) was 0.44 on its accuracy score. Cekura's voice agent workflow benchmark applies the same discipline to two open source frameworks: a scenario counts as passed only when all three retained runs pass. The LiveKit configuration passed all three retained runs on 70.73% of 82 scenarios and the Pipecat configuration on 63.41%. Each provider chose its own models and settings, and calls that failed to connect count against that pass³ figure. Pipecat's harness offers a repeat flag for exactly this reason. Cekura's GitHub Action takes a frequency input that runs each scenario N times and fails the workflow if any run fails, which is how regression testing for voice AI agents catches a flaky booking flow.
Should you build voice agent testing on open source tools or buy a platform?
Evaluate the open-source setup as a complete test workflow: the repository license, the runner, audio generation, scoring, and the place results are stored. Verify the license in the repository you actually install and pin the version used for the example. An open-source runner can still incur model, speech, telephony, and hosting costs; separate software licensing from the cost of a test run.
Building on open source tools is cheaper when one framework, text-level checks and an engineer's time are all a team needs. The LiveKit test framework and Pipecat's harness both slot into CI, and for tool-call assertions they are the right first layer. The build cost grows with coverage: a caller simulator that sounds like real callers, accents and background noise, phone and SIP routing, a judge calibrated against human labels, and a scenario library someone keeps current as the prompt changes. EVA-Bench needed a validation step that detects user-simulator errors and regenerates conversations before scoring, one component teams rarely budget for. Cekura covers those layers as a hosted service: phone, SIP and WebRTC connections, accent testing, multi-turn red-team scenarios, and a GitHub Action that gates a merge in CI/CD. The cost is the per-minute bill plus $30 a month for each seat after the first, traded against the engineering time a harness consumes.
What does Cekura give away for free, and what is actually open source?
Cekura is a commercial voice agent testing platform with a free starting tier and a small open source footprint. New accounts get 300 free credits, which the pricing page equates to about 60 minutes of voice testing, with no card required and one seat free. Cekura's Python metrics consume no credits at any volume, so deterministic checks on the transcript, call duration and end reason cost nothing to run. The open source part is the cekura-ai/metrics repository: seven Python modules under Apache-2.0, each building an LLM-judge prompt for a transcript, covering hallucination against a knowledge base, relevancy, sentiment, topic, drop-off point, call termination and casual chat. Teams can reuse those prompts in their own judge without an account. Cekura's benchmark runner is public on GitHub but carries no licence file, so it is source-available rather than open source. Cekura is one of the partner services LiveKit's testing documentation lists for end-to-end testing through the full audio pipeline.
Frequently asked questions
What is the best free tool for testing a voice agent end to end?
The best free end-to-end option depends on the stack. Pipecat teams should start with Pipecat's release-eval harness, whose caller voice, transcriber and judge run locally with no per-run cost, while the bot's own service keys still bill. LiveKit teams get text-based tests in pytest from the LiveKit Agents framework. For an agent reached by phone, Cekura's 300 free credits cover about 60 minutes of simulated calls over phone, SIP or WebRTC before any bill. The guide to choosing a voice AI testing platform sets out the wider criteria.
How do I add free voice agent tests to CI?
Text-level tests are the free CI layer. The LiveKit Agents test framework runs in pytest or Vitest, voicetest runs inside a GitHub workflow, and Pipecat's harness runs from a shell script. Audio tests in CI bill for whichever of the caller, the agent and the judge run on hosted models; Pipecat's local defaults leave only the bot's own service keys billing. Cekura's GitHub Action runs a scenario suite on push or pull request and fails the workflow if any run fails, billed at $0.25 per voice testing minute.
Do open source voice testing tools meet enterprise compliance and audit requirements?
Open source licences put the code in your hands, which suits teams that cannot send call audio to a third party, but the tools ship no audit trail, access control or signed agreements. The team running them builds and evidences those controls itself. Cekura's Enterprise plan lists SSO, SCIM, audit logs, VPC or on-premises deployment and a custom BAA and DPA, and its Startup plan includes a signed BAA and DPA.
Does Cekura handle free and open source voice agent testing?
Cekura is not open source, and it handles free voice agent testing in three checkable ways. New accounts get 300 free credits, about 60 minutes of voice testing, with no card and one free seat. Python metrics run at no credit cost. The cekura-ai/metrics repository publishes seven transcript-judge prompts under Apache-2.0. Beyond the free credits, Cekura bills $0.25 per voice testing minute, rates read 23 September 2026.
