Open Source Voice Agent Testing Tools

Open source voice agent testing tools are free, self-hosted projects for testing conversational voice agents. VoiceTest is the most purpose-built option for platform coverage; LangWatch Scenario is the strongest option for testing actual audio, though it doesn't yet cover every platform. Cekura scores both transcript accuracy and audio clarity as separate named metrics inside one product.

TL;DR

• Open source voice agent testing tools split into several categories: dedicated voice simulators, transcript-level LLM evaluation libraries, observability platforms, and framework-provided test utilities, each covering a different slice of the problem.
• VoiceTest is the most purpose-built open-source option, supporting Retell, Vapi, Bland, LiveKit, and Telnyx with autonomous multi-turn simulation and LLM-judge scoring.
• Most of the open-source stack tests at the transcript level, VoiceTest's own audio evaluation setting ships disabled by default, but LangWatch Scenario is a real exception that tests actual audio, with background noise and interruption simulation included.
• A typical open-source stack combines a simulator like VoiceTest with a regression tool like Promptfoo or DeepEval, an observability layer like Langfuse, and CI via GitHub Actions.
• Cekura scores voice clarity and jitter with a dedicated model that reads the actual audio channel, the layer the open-source stack consistently leaves untested.

What Do Open Source Voice Agent Testing Tools Actually Cover?

Open source voice agent testing tools fall into a few distinct categories, and confusing one for another is the fastest way to end up with a gap nobody planned for. Dedicated voice simulators, VoiceTest chief among them, run autonomous multi-turn phone conversations against a real agent and score the result with an LLM judge. Transcript-level evaluation libraries, Promptfoo and DeepEval, work one layer up: they test what a language model said, not how it sounded saying it. Observability platforms like Langfuse trace what happened in production but don't simulate anything before launch. Voice frameworks vary in how much testing they actually provide for a developer's own agent. LiveKit Agents includes a genuine built-in test pattern that runs a developer's own agent through a session and scores it with an LLM judge. Cekura's own Pipecat testing guide covers where Pipecat sits on this spectrum: Pipecat provides the primitives for observability, but no simulation or scenario-testing layer for the agent itself.

Which Tool Fits Which Job?

Matching a tool to the actual job matters more than picking the most popular one. A broader framework for evaluating a voice AI testing platform covers feature, integration, AI, and infrastructure layers, the same four questions worth asking of an open-source tool before adopting it, not only a commercial one. A team building on Retell or Vapi with a small test suite gets real value from VoiceTest alone. A team on ElevenLabs, OpenAI Realtime, or Pipecat that specifically needs real audio testing has LangWatch Scenario as a more audio-native option, provided its platform is one it actually supports. A team already deep in LLM evaluation tooling often adds DeepEval or Promptfoo for the transcript layer specifically, then still needs something else for the audio layer neither one touches.

Which Open Source Tools Actually Test Real Audio?

LangWatch Scenario is the clearest open-source exception that tests real audio. It injects real audio clips, simulates background noise and interruptions on the actual signal, and reports latency percentiles from real calls rather than estimates. Most of the rest of the open-source ecosystem grew up around transcripts instead, since text is easier to diff, store, and score than sound. DeepEval's repository lists dozens of metrics, RAG, agentic, multi-turn, none of them audio, voice, or telephony related. Even VoiceTest's own repository reflects the same default: it ships an audio evaluation setting, disabled out of the box, alongside its core transcript-and-judge workflow. Scenario's platform adapters cover a different, non-overlapping set from VoiceTest's, though, with LiveKit and Vapi both tracked as future work rather than shipped today. Cekura's own take on the observability gap makes a related point about tracing specifically: traces tell you a call happened, they don't confirm that the agent did the right thing. Cekura tests the audio layer directly across every supported platform, with a dedicated model that scores clarity and jitter from the audio channel itself, not a transcript reconstructed after the fact.

How Do You Combine Open Source Tools Into a Testing Stack?

No single open-source project covers simulation, regression, and observability at once, so most teams that go this route end up running three or four tools instead of one. A common pattern is VoiceTest for the initial simulated conversations, Promptfoo or DeepEval layered on top for prompt-level regression assertions, and Langfuse for tracing once the agent reaches production. GitHub Actions ties the pieces together, triggering the simulation and regression steps on every pull request. Cekura's own evaluation metrics cover a comparable set of signals, latency, talk ratio, interruption recovery, inside one product instead of three separately maintained ones. The tradeoff runs the other way too: an open-source stack costs nothing per seat, but it costs engineering time to keep three integrations working together.

How Do You Generate Enough Test Scenarios?

A handful of manually written test cases rarely survives contact with real callers, who ask things nobody scripted. IntellAgent, an open-source research framework, generates test scenarios automatically through policy-driven graph modeling instead of hand-writing every conversation, producing diagnostics on where an agent's coverage actually breaks down. This kind of automated generation matters more for voice agents than text-only ones, since a voice conversation has more ways to go wrong per turn: an interruption, a misheard word, a caller who changes their mind mid-sentence. Cekura's own guide to generating voice agent test cases covers the same problem, building a scenario set without hand-writing every conversation individually.

What Mistakes Do Teams Make When Adopting Open Source Voice Testing Tools?

Adopting an open-source testing stack goes wrong in a few consistent ways, most of them about assuming one tool covers more ground than it actually does.

• Assuming a transcript-level pass means the call sounded fine. A clean transcript can still represent a call with dead air, a robotic tone, or a background noise problem that never touches the text.
• Running one tool and assuming it covers the whole pipeline. VoiceTest, Promptfoo, and DeepEval each test a different slice; none of them alone is a complete voice-testing stack.
• Skipping load and concurrency testing entirely. Most open-source projects are built and tested against a handful of calls at a time, not the hundreds a real release needs validated at once.
• Assuming every framework provides the same amount of testing. LiveKit Agents ships a genuine judge-based pattern for testing a developer's own agent; Pipecat's own guide describes primitives for observability but no simulation or scenario-testing layer. Worth checking case by case rather than assuming.
• Never budgeting the integration time. Three or four separately maintained open-source tools cost real engineering hours to wire together and keep working, even at zero license cost.

Cekura's own audio-native scoring and load testing exist specifically to close the gaps the first three mistakes create.

FAQ

What's the best open source tool for testing voice agents?

It depends on whether platform coverage or real audio testing matters more. VoiceTest is the most purpose-built option for platform coverage, supporting autonomous multi-turn simulation against Retell, Vapi, Bland, LiveKit, and Telnyx agents with LLM-judge scoring, though it tests at the transcript level by default with its audio evaluation setting shipped disabled. LangWatch Scenario is the stronger pick if testing actual audio matters more than platform breadth, on the platforms it currently supports. General LLM evaluation libraries like Promptfoo and DeepEval are strong complements for prompt-level regression but were not built for voice specifically.

Can open source tools test audio quality, not just transcripts?

Mostly not by default, but there's a real exception. VoiceTest ships an audio evaluation option that's disabled out of the box, and DeepEval has no audio, voice, or telephony metrics at all. LangWatch Scenario does test real audio, injecting clips, simulating background noise, and measuring actual latency, though its platform adapters don't yet cover LiveKit or Vapi. For broad platform coverage with audio testing included by default, that currently means a platform built for it specifically.

Is Langfuse a voice agent testing tool?

No. Langfuse is an observability and tracing platform: it records what happened during a production conversation but doesn't simulate conversations before launch or judge whether the agent behaved correctly. It's commonly paired with a testing tool like VoiceTest rather than used as a substitute for one.

How do you build a complete open source testing stack for a voice agent?

Combine a dedicated simulator like VoiceTest for conversation-level testing, a transcript regression tool like Promptfoo or DeepEval for prompt-level assertions, and an observability platform like Langfuse for production tracing, then wire the simulation and regression steps into CI with GitHub Actions. No single open-source project currently covers all three layers at once.

Does Cekura test the audio itself, not just the transcript?

Yes. Cekura scores voice clarity and jitter with a dedicated model that reads an agent's actual audio channel, the same layer most open-source tools either skip or ship disabled by default. Transcription accuracy is tracked as its own separate metric alongside it, so a wording issue and an audio issue never get folded into one blended score.