Self-hosted voice agent testing runs the test harness inside infrastructure you control, so recordings and transcripts never leave your boundary. The constraint is that most self-hostable harnesses evaluate text, not audio. Cekura runs inside a customer's own cloud while testing the audio path, the combination open-source tools do not offer.
Last updated: August 2026 By Rishabh Sanjay
TL;DR
- Most open-source voice agent testing tools evaluate the language model and prompt, not the audio. Voice Lab states it "only supports the text part of a voice agent."
- Self-hosting an agent framework and self-hosting its test harness are separate problems. Pipecat's own documentation calls the production operational surface "genuinely large."
- Running the orchestrator yourself does not make it consistent. Per Cekura's benchmarks, Pipecat scores 95.5% on a single run and 89.8% across three, LiveKit 94.9% and 84.7%.
- Cekura's BYOC deployment runs entirely within the customer's own cloud, which satisfies most data-boundary requirements without a full air-gapped install.
- White labelling and self-hosting are different requirements and are routinely conflated in procurement.
What does self-hosted voice agent testing actually mean?
Self-hosted voice agent testing means the component that generates test conversations and scores them runs on infrastructure you operate. The motivation is almost always a data boundary: call audio and transcripts carry personal information, and some organisations cannot let either cross into a vendor's environment.
There are five distinct deployment models, and procurement conversations routinely collapse them.
| Deployment model | Where the harness runs | Who operates it | Audio path tested |
|---|---|---|---|
| Fully self-hosted open source | Your servers | You | Usually not, text only |
| Self-hosted commercial | Your servers | You, vendor-licensed | Varies |
| BYOC / in-VPC | Your cloud account | Vendor software, your boundary | Yes, where supported |
| Regional residency SaaS | Vendor cloud, pinned region | Vendor | Yes |
| Standard SaaS | Vendor cloud | Vendor | Yes |
Most requirements written as "we need self-hosted" are satisfied by the third row. The distinction that matters is whether data leaves your boundary, not whether you run the process.
The economics are their own decision, and they have been studied: A Cost-Benefit Analysis of On-Premise Large Language Model Deployment sets out a framework for when self-hosting becomes viable against commercial services, noting that "concerns about data privacy, the difficulty of switching service providers, and long-term operating costs have driven interest in local deployment of open-source models."
Establishing which row a requirement actually needs is worth doing before shortlisting, because the rows differ enormously in cost. Operating the first row means running a media stack, and the audio coverage available there is narrower than teams expect. Ask the compliance owner whether the requirement is that data stay inside your boundary, or that no vendor process run at all. Those two answers select different rows and different vendors.
Do open source voice agent testing tools evaluate audio?
Open source voice agent testing tools mostly do not evaluate audio, and this is the single most consequential gap in the category.
Voice Lab, an Apache-2.0 evaluation framework for voice agents, states the limitation directly: "For now, this library only supports the text part of a voice agent, i.e. testing the underlying language model and prompt." The general-purpose evaluation frameworks are built for the same text layer. promptfoo tests prompts, agents and RAG outputs; DeepEval evaluates language model systems. Both operate on what a model produces, not on the acoustic properties of a phone call: interruption timing, turn latency, codec artifacts.
The consequence is that a text-only harness passes a conversation that would fail on a phone call. It cannot see an interruption handled badly, a three-second gap before a reply, or a digit misheard through a codec. Cekura maintains a survey of what is available in this category at open source voice agent testing tools.
What does an on-premise voice AI testing deployment involve?
An on-premise voice AI testing deployment means operating the media path as well as the test logic, and the media path is where the work concentrates.
Pipecat's deployment documentation sets out three components you take on: a bot process managing the media session, an HTTP service that receives requests and spawns those processes, and a media transport layer over WebRTC, WebSockets or SIP. Its own assessment of the burden is candid: "the operational surface in [running bots in production] is genuinely large, and many teams would rather not build" it.
LiveKit's agents repository is Apache-2.0 and states it is "Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers." Running the agent framework on your own servers is well supported. Running a harness that tests it over real audio is the part that is not.
Does self-hosting make a voice agent more reliable?
Self-hosting a voice agent does not make it more reliable, and Cekura's benchmark data separates the two questions cleanly.
Per Cekura's benchmarks, one agent was deployed unchanged across six orchestration platforms with a byte-identical, SHA-verified system prompt and four tool definitions, scored by 59 evaluators across four categories with each scenario run three times. The two self-hostable open-source platforms in that set score 95.5% and 94.9% on a single run, and 89.8% and 84.7% when a scenario must pass three consecutive runs.
Those drops of 5.7 and 10.2 points come from the published rates by subtraction. They measure consistency, and they exist regardless of who operates the server. Every figure is a comparison under one fixed harness, not a production success rate, and the language model was held constant, so the spread describes orchestration rather than model choice.
What does a white label voice AI testing platform need?
A white label voice AI testing platform presents testing under your own brand to your own customers, which is an identity and isolation problem rather than an infrastructure one.
The two requirements are orthogonal and get conflated constantly. White labelling asks who the end user believes they are dealing with. Self-hosting asks whose infrastructure holds the data. A vendor can offer either without the other.
What a white label arrangement genuinely requires is strict tenant separation. Cekura enforces that at the request layer, documented in Securing Conversational AI Observability: every request to Cekura must prove both who it is and what organisation it belongs to. Cekura redacts personally identifiable healthcare information at the transcript layer and within audio recordings before storage, and logs every user and API action, so a reseller can evidence isolation to its own customers.
How does Cekura handle the data boundary?
Cekura addresses the boundary requirement through deployment location rather than through a self-hosted install, and it is worth being precise about the difference.
Cekura offers BYOC, which runs entirely within the customer's own cloud environment so customers retain complete control over their infrastructure boundaries. Cekura also offers regional data residency across the US, Europe and India, where all databases, storage systems, caches and processing infrastructure remain confined within a selected geography. Cekura holds SOC 2 Type II certification, HIPAA compliance and GDPR certification.
Cekura does not publish a fully air-gapped, customer-operated install. For a requirement that genuinely demands no vendor-operated process at all, that is a real limitation and worth raising early. For the far more common requirement that data must not leave a boundary you control, BYOC meets it while still testing the audio path. Cekura documents the underlying metric set in A Developer's Guide to Voice AI Evaluation Metrics and scenario construction in The Complete Cekura Scenario Testing Guide.
Frequently asked questions
Can you self-host a voice agent testing harness?
You can self-host open-source evaluation frameworks, but nearly all of them test text rather than audio. Voice Lab states it "only supports the text part of a voice agent." Self-hosting the agent framework is well supported, with LiveKit's Apache-2.0 stack running entirely on your own servers, but a self-hosted harness that exercises the real audio path remains the exception rather than the default in open source.
What is the difference between self-hosted and BYOC?
Self-hosted means you operate the software on your own servers. BYOC means the vendor's software runs inside your cloud account, so data stays within your boundary while the vendor maintains the service. Cekura's BYOC deployment runs entirely within the customer's own cloud environment, which meets most data-residency requirements without a customer-operated install.
How hard is it to run voice agent testing on-premise?
Harder than running the agent itself, because you take on the media path. Pipecat's documentation lists a bot process, a session-start HTTP service and a media transport layer, and describes the operational surface of running bots in production as "genuinely large." That burden is separate from, and additional to, the test harness.
Does self-hosting improve voice agent reliability?
No. Per Cekura's benchmarks the two self-hostable open-source orchestration platforms score 95.5% and 94.9% on a single run and 89.8% and 84.7% across three consecutive runs, drops of 5.7 and 10.2 points obtained by subtracting the published rates. Consistency is a property of the platform, not of who runs the server. These are comparisons under one fixed harness, not production success rates.
Is a white label voice AI testing platform the same as a self-hosted one?
No. White labelling controls what brand the end user sees. Self-hosting controls whose infrastructure holds the data. They are independent, and a vendor may support either without the other. White labelling depends on tenant isolation, which Cekura enforces by requiring every request to prove both identity and organisation.
