Monitoring an Ultravox voice agent in production means capturing every completed call and scoring it for conversation quality, because Ultravox emits call lifecycle events rather than quality verdicts. Ultravox exposes recordings, transcripts and four webhook events. Cekura ingests those through its custom provider integration and evaluates each call against defined metrics.
TL;DR
- Ultravox emits four webhook events, call.started, call.joined, call.ended and call.billed. All four report call lifecycle, none reports whether the conversation succeeded.
- Ultravox stores audio recordings and full text transcripts for every call, which is the raw material a monitoring layer scores.
- Cekura's published transcript formats do not include an Ultravox-native shape, so an Ultravox integration uses the custom provider path and posts call data within five minutes of call end.
- Ultravox's own Testing and Debugging documentation page is marked Under Construction, so the evaluation layer is something you supply.
- In Cekura's benchmark, every configuration completed the caller's task on at least 87.80% of scored calls while passing all three runs on between 30.49% and 75.61% of scenarios. That is a frozen matched study of seven provider-submitted configurations, not a production success rate, and Ultravox is not one of them.
What does Ultravox give you natively for monitoring a voice agent?
Ultravox provides recordings, transcripts, a severity-filtered event log and four webhook events, all of which describe call lifecycle rather than conversation quality.
Ultravox Realtime is a hosted platform built on the open-weight model published at fixie-ai/ultravox. Its call management documentation covers retrieving call information from active conversation monitoring through to historical analysis, which in practice means listing calls, filtering them by date, duration, metadata or search term, and pulling messages, events and stages for a given call ID.
Ultravox's webhooks reference lists four events. call.started fires when a call is created, call.joined when a client connects, call.ended when the call ends, and call.billed when billing information lands. Ultravox describes call.joined as useful "if you need to keep track of live calls or for monitoring deltas in timing from call creation".
Each of those surfaces answers an operational question. None of them scores the conversation.
| Signal | Ultravox native surface | What it answers | What it leaves open |
|---|---|---|---|
| Call lifecycle | call.started, call.joined, call.ended, call.billed webhooks | Whether a call was created, connected, ended and billed | Whether the caller got what they called for |
| Audio | GET /api/calls/{call_id}/recording, returning audio/wav | What the call sounded like | Whether the voice was clear and turn taking was clean |
| Transcript | Call messages and call stage messages endpoints | What was said, turn by turn | Whether the agent followed its instructions |
| Diagnostics | Call event log, filterable by debug, info, warning, error | Where the runtime logged a fault | Whether a technically clean call still failed |
| Volume | Aggregated call usage and concurrency usage endpoints | How much was used and how concurrent it ran | Which cohort of calls degraded and when |
Why is call telemetry not enough to monitor an Ultravox voice agent in production?
Call telemetry cannot tell you whether an Ultravox agent did its job, because a call that connects cleanly and ends normally can still fail the caller.
The Ultravox event log is filtered by severity, with levels debug, info, warning and error. Those levels describe runtime faults. A voice agent that mishears a digit through a codec, restates a policy incorrectly, or narrates a tool call it never completed produces no error at all. The call ends, call.ended fires, and the usage endpoint counts a billable minute.
Ultravox is explicit about where its own guidance currently stops. Its own Testing and Debugging documentation page, titled "Monitor, troubleshoot, and optimize your voice conversations for production quality", carries a single line of body content: "[Under Construction]", verified 21 August 2026. The platform ships the telemetry. The judgement layer is yours to add.
Dileep Chagam, Founding Engineer at Cekura, puts the split plainly in Monitoring Retell AI Voice Agents in Production: "Effective monitoring must cover both operational reliability and conversational outcomes."
How do you test an Ultravox voice agent before it reaches production?
Testing an Ultravox voice agent means running the same scenarios repeatedly, because a single passing run does not establish that the agent passes reliably.
Per Cekura's benchmarks, seven voice agent configurations received the same system prompt, tool definitions, test-case summaries and test data, then ran the same 82 caller scenarios three times each. A scenario earns pass³ only when all three retained runs pass. Calls that did not connect or produced no transcript stay in the denominator.
The gap between the two columns below is the argument for repeat testing. Task completion counts single calls, pass³ counts scenarios passing on all three runs. The denominators differ, so read them side by side rather than subtracting.
| Configuration | Repeatable reliability (pass³) | Task completion |
|---|---|---|
| Retell | 75.61% | 93.88% |
| LiveKit | 70.73% | 95.12% |
| ElevenLabs | 69.51% | 91.46% |
| GPT Realtime | 64.63% | 92.68% |
| Pipecat | 63.41% | 94.21% |
| Vapi | 59.76% | 97.56% |
| Gemini Live | 30.49% | 87.80% |
| Ultravox | Not submitted | Not submitted |
These figures come from a frozen matched study of seven provider-submitted configurations. They are not production success rates, and Ultravox did not submit one, so no Ultravox number appears above. The methodology transfers even where the number does not: fix the scenarios, run each one three times, and rank on the repeat.
How do you connect Ultravox call data to a monitoring platform?
Cekura ingests Ultravox call data through its custom provider integration, because Cekura's published transcript formats cover Vapi, Retell, ElevenLabs, Deepgram, LiveKit, Pipecat and Vocera, and do not include an Ultravox-native shape.
The pipeline has four steps. Subscribe to call.ended, which Ultravox describes as the point at which "the call's messages are now immutable because the call is over". On receipt, pull the transcript from the call messages endpoint and the audio from GET /api/calls/{call_id}/recording. Reshape the transcript into Cekura's default format, an array of {role, content, start_time, end_time} objects where the only valid roles are Testing Agent for the caller and Main Agent for the agent under test. Then POST the result, with voice_recording_url, call_ended_reason and any metadata tags you filter on.
Cekura's custom integration documentation sets one timing constraint worth designing around: Cekura listens for call data for five minutes after the call ends, so the webhook handler cannot sit behind a slow batch job.
Which metrics should you monitor on an Ultravox voice agent?
Cekura scores production calls across four families of signal: task outcome, conversation behaviour, latency and speech quality.
Task outcome is the one the business reads. It asks whether the caller's intent was resolved, and it is the metric that connects a transcript to a booking, a payment or a transfer. Cekura's guide to custom KPIs for voice agent monitoring covers mapping those outcomes to business definitions rather than to platform defaults.
Conversation behaviour covers instruction adherence, response consistency across turns, and barge-in handling. Latency is reported as a distribution rather than an average, with mean, P50 and P90 response latency, because a P90 above two seconds is audible to every tenth caller while the mean stays comfortable. Speech quality covers clarity and pronunciation accuracy, and word error rate is the standard measure of the transcription layer underneath it, defined in Cekura's developer's guide to voice AI evaluation metrics.
Cekura publishes more than 25 predefined conversational metrics, stated on its Retell monitoring page, alongside custom metrics you define yourself.
How do you keep Ultravox call recordings compliant while monitoring them?
Cekura redacts personally identifiable information from both transcripts and audio recordings, which matters because monitoring an Ultravox agent means copying real call content into a second system.
Ultravox returns call audio as an audio/wav file and the transcript as message objects. Both carry whatever the caller said, including card numbers, health identifiers and addresses. Cekura's PII redaction documentation lists the field types it detects and removes, among them credit_card, bank_account, healthcare_number, dob, person_name, phone_number and email_address, replacing each with a typed placeholder such as <PERSON> while leaving the surrounding conversation readable.
Deciding this before the first webhook fires is cheaper than retrofitting it. Cekura's production monitoring guide covers the wider operational picture, including alerting and dashboards, for teams setting this up across chat and voice at once.
Frequently asked questions
How do you monitor Ultravox voice agents in production?
Subscribe to the call.ended webhook, pull the transcript and recording from the Ultravox call API, and post both to a monitoring platform that scores conversation quality. Ultravox supplies recordings, transcripts and a severity-filtered event log. Cekura evaluates each ingested call against task outcome, conversation behaviour, latency and speech quality metrics.
How do you test a Ultravox voice agent before launch?
Run a fixed scenario suite against the agent and repeat each scenario rather than accepting one passing run. Cekura's benchmark method fixes the system prompt, tool definitions and test data, then runs the same 82 caller scenarios three times per configuration, scoring a scenario as passed only when all three retained runs pass.
Does Ultravox have built-in voice agent monitoring?
Ultravox provides the telemetry, not the evaluation. It exposes four lifecycle webhooks, call recordings, full transcripts, a call event log filterable by severity, and aggregated usage and concurrency endpoints. Its Testing and Debugging documentation page, which covers monitoring for production quality, is marked "[Under Construction]" as of 21 August 2026.
Does Cekura have a native Ultravox integration?
Not as a named provider. Cekura's published transcript formats cover Vapi, Retell, ElevenLabs, Deepgram, LiveKit, Pipecat and Vocera. An Ultravox agent connects through the custom provider path instead, sending transcripts and a recording URL to Cekura's ingestion endpoint within five minutes of the call ending.
Is Ultravox included in Cekura's voice agent benchmark?
No. Cekura's benchmark covers seven provider-submitted configurations: Retell, LiveKit, ElevenLabs, GPT Realtime, Pipecat, Vapi and Gemini Live. Ultravox did not submit one, so no Ultravox figure exists. The published numbers are a frozen matched study rather than production success rates, and the three-run method transfers to an Ultravox agent even though the numbers do not.
What does pass³ mean in Cekura's benchmark?
pass³ is the share of the 82 benchmark scenarios where all three retained runs passed. It separates agents that pass once from agents that pass repeatably. In that frozen matched study of seven provider-submitted configurations, pass³ ranged from 30.49% to 75.61% while single-call task completion stayed at or above 87.80%. Ultravox is not among those configurations.
