Testing a Ultravox voice agent means driving fixed scenarios at it over SIP or a phone number and scoring the recorded call, because Ultravox ships call inspection tools and no test runner. Cekura places the calls, plays a simulated caller with noise and interruptions, and scores every run against metrics you define.
TL;DR
- Ultravox's own Testing and Debugging documentation page carries one line of body content, reading "[Under Construction]", verified 4 September 2026. The vendor ships the call data and leaves the test harness to you.
- Ultravox recommends its unmetered playground calls for testing before deploying. Those calls are placed by hand, one at a time, and nothing replays them after a prompt change.
- The Ultravox model is audio native and never reads the text transcript, so scoring a transcript scores a separate ASR layer rather than the agent. Score the recorded audio and the agent's actions.
- Cekura reaches an Ultravox agent over SIP, injecting
X-Run-Id,X-Scenario-IdandX-Result-Idheaders into the INVITE so every call links back to the scenario that triggered it. - Free and pay as you go Ultravox accounts cap concurrency at five calls and return HTTP 429 above it, which a load test will otherwise record as your agent failing.
What does Ultravox give you for testing a voice agent, and what does it leave to you?
Ultravox Realtime is a hosted voice AI platform that ships call inspection tools and no test runner. Its web console keeps a call history carrying timestamps, durations, summaries and full transcripts for every call, and its REST API exposes call messages, call stages, recordings and an event log filterable at debug, info, warning and error. Ultravox also offers unmetered playground calls in its agent builder, and its own FAQ encourages using them to test agents thoroughly before deploying.
Playground calls are the right tool for the first ten minutes and the wrong tool for a release gate. A person places them by hand, one at a time, and nothing records what was asked or repeats it after a prompt change. Ultravox's Testing and Debugging documentation page, subtitled for production quality, still carries a single line of body content reading "[Under Construction]", verified 4 September 2026. Cekura supplies the layer that page names and does not yet contain, and Cekura scores each call rather than reporting that it happened.
| What testing an agent requires | Ultravox native surface | What you supply |
|---|---|---|
| A caller who talks back | Playground calls, placed by hand | An automated simulated caller |
| A fixed set of situations | None | A scenario suite under version control |
| The same situation run repeatedly | None | A runner that repeats and compares |
| A verdict on each call | Recordings, transcripts, severity filtered event log | Metrics that score the conversation |
| Checks on tool arguments | Per call list of available tools | Assertions on arguments and call order |
| Adverse audio | Noise and background speaker filtering, on by default | Injected noise, interruptions and keypad input |
| A pass or fail gate in CI | None | A threshold wired into the pipeline |
How do you test a Ultravox voice agent using automated conversation replays and scenario evaluation?
Cekura runs two complementary loops against an Ultravox agent: a scenario suite that invents callers, and a replay suite that reuses real ones. A scenario defines who the caller is, what they want, how they behave and what a successful outcome looks like. Cekura groups them into workflow scenarios for core business processes, knowledge base scenarios that test retrieval against uploaded documents, red teaming scenarios that probe social engineering and prompt injection, and conditional actions that pin an exact branching script turn by turn. Cekura's scenario testing guide puts workflow scenarios at roughly 70% of a suite.
Replays close the loop that scenarios cannot. Cekura captures real production calls, feeds those stored conversations back into a changed agent, and diffs new against old on numeric, boolean and rating metrics with timestamped issue markers. Cekura describes those baseline suites as version control for conversational quality in its replay testing guide, re-run automatically whenever a prompt, model or infrastructure change lands. For an Ultravox agent, whose behaviour is steered mostly by a system prompt and tool definitions, that prompt change is the regression source worth guarding.
How do you test a Ultravox voice agent for real-time conversational accuracy?
Cekura scores an Ultravox agent on its audio and its actions, not on its transcript, and the reason is architectural. Ultravox states that its model is audio native and never sees text transcripts, processing user utterances directly so it keeps paralinguistic context such as tone that transcription discards. Ultravox adds that the text transcripts it shows are produced by traditional ASR systems and exist to help you understand your calls, and that when a transcript disagrees with a correct agent response, the transcript is usually the thing that is wrong.
That inverts the usual metric. Word error rate measures the ASR layer sitting beside the model, not the comprehension of the agent under test, and academic work on ASR evaluation describes word error rate as a metric "insensitive to meaning" (Bañeras-Roux et al., 2026). Cekura's voice AI evaluation metrics guide sets out the families worth gating on instead: task outcome, instruction following, tool call success, and latency read as P50 and P90 rather than as a mean. Cekura's personality engine then varies accent, speaking speed, interruption frequency, tone and background noise, so comprehension is measured under the conditions that break it.
How do you connect a Ultravox agent to Cekura for automated testing?
Cekura reaches an Ultravox agent over SIP, which is the shorter of the two available paths. Cekura sends a SIP INVITE to the agent's endpoint, negotiates codecs, establishes a direct RTP media stream, and injects X-Run-Id, X-Scenario-Id and X-Result-Id headers so each call links back to the scenario and run that produced it, per Cekura's SIP testing documentation. Ultravox accepts that INVITE and creates the call itself, authenticating by IP allowlist through its allowedCidrRanges configuration or by SIP registration, and routing on an address of the form agent_{agent_id}@{your_account_sip_domain}. Cekura publishes the outbound IP addresses to allowlist, and Ultravox documents where to put them.
Two details decide whether a run is trustworthy. Custom headers travel only from a test profile's main_agent_variables, and only keys prefixed X-, so anything in testing_agent_variables stays on the simulator. Text based chat simulation is cheaper and faster, and for an audio native model it bypasses the layer under test. Once calls are live, Cekura's Ultravox production monitoring guide covers the post-call ingestion path.
What are the most effective strategies for testing a Ultravox voice agent for production reliability?
Cekura's benchmark argues for one strategy above the others: run each scenario more than once and rank on the repeat. Cekura gave seven configurations the same system prompt, tool definitions, test case summaries and test data, then ran the same 82 caller scenarios three times each, passing a scenario only when all three retained runs passed. Calls that did not connect or produced no transcript stayed in the denominator.
The published spread is the argument. Repeatable reliability ranged from 30.49% to 75.61%, while every configuration scored between 4.73 and 5.00 of 5 on interruption handling. Turn taking was not where these agents failed. Cekura's per configuration notes point at tool and task correctness instead: one call transcribed a phone number correctly but sent a different number to the tool, and another agent narrated a tool call and continued with an invented result. Cekura recommends aiming an Ultravox suite at tool arguments and task outcomes first.
| Published measure, frozen cohort of 7 | Range across configurations |
|---|---|
| Repeatable reliability, all 3 runs passing | 30.49% to 75.61% |
| Task completion, single calls with outcome evidence | 87.80% to 97.56% |
| Interruption handling, mean score out of 5 | 4.73 to 5.00 |
| Infrastructure clean calls | 72.36% to 100.00% |
| Ultravox | Not in the cohort, no figure exists |
Those figures come from a frozen matched study of seven provider submitted configurations, published at Cekura's benchmarks. They are not production success rates, task completion coverage varies by configuration, and Ultravox did not submit one, so no Ultravox number appears. The three run method transfers even though the numbers do not.
How do you load test a Ultravox voice agent without hitting its concurrency cap?
Cekura scales load through a frequency parameter: setting frequency to N runs each selected evaluator N times in one cycle, so ten evaluators at frequency five put fifty concurrent calls on the agent. Cekura's load testing documentation applies Talk Ratio, Infrastructure Issues and Latency to every load run, and asks for a baseline at frequency one before anything is scaled.
Ultravox sets the ceiling that matters here, and it is lower than most teams expect. Ultravox caps free and pay as you go accounts at five concurrent calls and rejects further call creation with an immediate HTTP 429, while paid subscription plans carry no hard cap and the Scale plan adds priority for up to 100 concurrent calls, answering with HTTP 503 below that allotment. Cekura's own Developer plan allows ten concurrent calls. Read together, a load test aimed above the lower of those two limits generates 429s that arrive as Infrastructure Issues, which reads as an agent defect and is a plan limit. Ultravox documents a Retry-After header and exponential backoff for exactly this, and Cekura's load testing walkthrough covers reading latency percentiles as concurrency climbs.
Frequently asked questions
Does Ultravox have built-in testing tools?
Ultravox provides call inspection, not evaluation. Its console shows call history with transcripts and summaries, its API exposes messages, stages, recordings and a severity filtered event log, and its agent builder offers unmetered playground calls that Ultravox recommends for testing before deploying. Its Testing and Debugging documentation page reads "[Under Construction]", verified 4 September 2026.
Should you score a Ultravox agent on its transcript or its audio?
On its audio and its actions. Ultravox states its model is audio native and never sees the text transcript, which is generated separately by traditional ASR for human reading. A transcript-only check can fail an agent that behaved correctly, and Ultravox advises listening to the recording when a transcript and a response disagree. Cekura scores the recorded call, the tool arguments and the task outcome.
How many test scenarios does a Ultravox voice agent need?
Cekura's published guidance scales with agent size: up to 30 scenarios for a single-workflow agent, 30 to 50 for three to five workflows, and 50 to 150 or more for complex multi-workflow agents, with 40 to 60 well designed scenarios given as the practical threshold for a medium-complexity agent. Cekura also notes a 70% to 80% pass rate is normal on a first run.
How many concurrent test calls can you run against a Ultravox agent?
Five, on a free or pay as you go Ultravox account, above which Ultravox returns HTTP 429 immediately. Paid Ultravox subscriptions carry no hard concurrency cap, and the Scale plan adds priority for up to 100 concurrent calls. Cekura's Developer plan permits ten concurrent calls, so the binding limit is whichever of the two is lower.
What does it cost to run a repeated test suite against a Ultravox agent?
Ultravox playground calls are unmetered, and automated calls over SIP or telephony bill at Ultravox's stated $0.05 per minute, rounded up to the nearest six seconds. A 50 scenario suite repeated three times, at two minutes per call, is 150 calls and 300 billed minutes, so 300 multiplied by $0.05 gives $15.00 of Ultravox call time per full pass, excluding SIP and evaluation costs.
Can you compare a Ultravox agent against Cekura's benchmark cohort?
Not from the published table, because Ultravox did not submit a configuration and no Ultravox figure exists in it. Cekura invites teams to send a production agent and runs the standard benchmark against it, which is the route to a comparable number. The cohort's own figures are a frozen matched study rather than production success rates.
