Voice AI agents built on Twilio ConversationRelay are tested by placing real calls into the agent's Twilio phone number, scoring each conversation against per-layer evaluators, and gating releases on pass rate and latency percentiles. Cekura places those calls using imported Twilio numbers, evaluates the transcripts, and monitors production traffic.
TL;DR
- ConversationRelay splits the agent across a boundary: Twilio runs speech-to-text, text-to-speech and the media path, and your WebSocket server runs the language model and the tool calls. Defects land on either side of that split, and neither side sees both.
- Twilio's Conversation Relay Insights dashboard reports time to first audio, interruption rate and error codes for calls that already happened. It cannot tell you whether the change you are about to ship breaks a booked flow.
- Conversation Intelligence (classic) stores utterances rather than audio, runs only after the call ends, and does nothing unless you set the intelligenceService attribute on the TwiML noun.
- Twilio publishes median latency under 0.5 seconds and 95th-percentile latency under 0.725 seconds for ConversationRelay, described on its own product page as based on internal benchmarks with results that may vary.
- Cekura places calls into the agent's Twilio number, scores each turn against per-layer evaluators, and gates a release on repeated passes rather than on a manual listen.
What makes testing a Twilio ConversationRelay agent different?
Testing a Twilio ConversationRelay agent is different because the agent is split across a network boundary, and each side sees only half of the call.
The <ConversationRelay> noun sits inside <Connect> and opens a WebSocket to your server. Twilio runs speech recognition, speech synthesis and the media path. Your server runs the language model and the tool calls. Neither half holds the full record of what happened.
Per Twilio's WebSocket message reference, Twilio sends setup on connect, prompt when the caller says something, dtmf on keypresses, interrupt when the caller talks over playback, and error on session faults. Your server replies with text tokens, play media, send digits, switch language and end session messages.
The interrupt message is the one most teams discard. It carries utteranceUntilInterrupt and durationUntilInterruptMs, which record what the agent had already said and how long synthesis continued after the caller started speaking. Cekura measures that same overrun independently, capturing how long TTS continued after user speech began.
Which tools test voice AI agents built on Twilio ConversationRelay?
Four tools and one manual practice cover voice AI agents built on Twilio ConversationRelay, and they separate by when they run and what they are able to see. No single one of them spans both sides of the WebSocket boundary.
| Tool | What it exercises | What it cannot see | When it runs |
|---|---|---|---|
| Manual test calls over an ngrok tunnel | End-to-end behaviour on whichever path you happen to try | Anything you did not think to try, and it produces no pass rate | Before release, ad hoc |
| Unit tests on your own WebSocket server | TwiML generation, message handling and tool-call code | The audio path, transcription errors, barge-in, task outcome | In CI, on every commit |
| Conversation Relay Insights | Time to first audio, interruption rate, silent calls, top error codes, turns and tokens per call | Whether the caller's task actually succeeded, and anything before release | After calls run in production |
| Conversation Intelligence (classic) | Stored utterances, sentiment, summarisation, and operators for task completion, escalation and hallucination | Audio, and anything before the call has ended | Post-call, and only when intelligenceService is set |
| Cekura | Simulated calls into the Twilio number, per-layer evaluators, repeated regression runs, production monitoring | Provider-native component timing inside Twilio's own stack | Before release and in production |
The split that matters is not feature depth, it is direction. Twilio's two observability products describe calls that already happened, which is the right shape for an incident review and the wrong shape for a release gate. Unit tests run early but exercise your server code rather than audio, so they will not catch a digit misheard through a codec or an agent that talks straight through an interruption.
Cekura sits on the missing axis. Cekura places calls into the agent before a change ships, runs the same caller situations again afterwards, and reports which scenarios stopped passing.
How to test a Twilio ConversationRelay voice agent?
You test a Twilio ConversationRelay voice agent by dialling it like a caller, because the agent is already fronted by a phone number.
Cekura states the connection options directly: "If the agent is fronted by a phone number, Cekura places inbound and outbound calls using imported Twilio or Plivo numbers." A ConversationRelay agent takes that path by construction: the Twilio number pointed at your TwiML endpoint is what makes it answerable. Cekura documents the SIP and webhook alternatives in Tools to Test Voice AI Agents Built on the Deepgram Voice Agent API.
Five steps take an agent from reachable to release-gated:
- Point the Twilio number at the TwiML endpoint that returns your <Connect><ConversationRelay> response.
- Import that number into Cekura so scenarios dial the real agent.
- Write scenarios as caller situations rather than scripts.
- Attach evaluators per layer: transcription, turn-taking, tool calls, task outcome, voice quality.
- Run each scenario more than once and gate the release on how often all runs pass.
Why should each test scenario run more than once?
Each test scenario should run more than once because a single pass proves the agent can succeed, not that it will.
Cekura's benchmarks run 82 scenarios three times against each of seven platform configurations and rank on pass³, the share of scenarios where all three retained runs passed. Calls that did not connect or produced no transcript stay in the denominator rather than being removed. Twilio ConversationRelay is not one of those seven configurations, so the method transfers and the numbers do not.
The cohort shows why one number misleads. Vapi records the highest task completion rate in the set, 97.56% of calls with outcome evidence, and still ranks sixth of seven on repeatable reliability at 59.76% pass³, because 41 of its 246 retained calls never connected. Cekura measures response time at the main-agent layer in that study, not provider-reported component timing. A team reading only task completion would have shipped.
How to monitor Twilio ConversationRelay voice agents in production?
You monitor Twilio ConversationRelay voice agents in production with two Twilio products, one for telemetry and one for transcripts.
Conversation Relay Insights reports average time to first audio, calls exceeding a 1.2 second threshold, interruption rate, silent calls, words per token and top error codes, and decomposes response time into application, network, speech-to-text and text-to-speech components. Twilio states three limits: it "is not a HIPAA Eligible Service", its latency excludes the last mile between the end user and Twilio's media edge, and its speech-to-text latency accuracy varies by model and language, within 100 ms for English and up to 250 ms elsewhere. Twilio adds that these "are not guarantees of performance".
Conversation Intelligence (classic) adds transcript persistence and Language Operators for task completion, escalation, hallucination and toxicity. Three constraints govern it. Transcripts are utterances, not audio. Analysis runs only after the call ends. Operators do not run when the session language differs from the Intelligence Service language, leaving the transcript stored and unscored.
Why can't Twilio's published latency numbers tell you the agent works?
Twilio's published latency numbers describe Twilio's half of the call, and the half you wrote is where task failures live.
Twilio's product page claims median latency under 0.5 seconds and 95th-percentile latency under 0.725 seconds for ConversationRelay, footnoted as based on internal benchmarks with results that may vary. Treat that as a floor on the transport rather than a statement about your agent. Your model call, your tool calls and your retrieval step sit on your WebSocket server, outside that measurement.
Dan Bartlett, who writes on voice AI for Twilio, puts the general case plainly:
"Voice AI success depends on business outcomes, not just technical performance. Latency and word error rate matter, but should be viewed as means to an end, not goals in and of themselves."
An agent can answer inside Twilio's published median and still take the wrong booking, and no percentile on the dashboard will say so. Cekura compares the wider platform landscape in Twilio vs Vapi.
How does Cekura test and monitor a ConversationRelay agent?
Cekura tests and monitors a ConversationRelay agent on the path a customer uses, with one evaluator set covering both.
Cekura dials the agent's Twilio number, runs caller situations that vary wording, accent, interruption timing and background noise, scores each conversation against per-layer evaluators, and reports which scenarios stopped passing after a prompt or model change.
In production Cekura scores live traffic against that same evaluator set, so a regression caught in staging and one caught in production are the same measurement. Cekura monitors hallucination, relevancy, latency, tool-call success, interruption and overtalk patterns, and stop time after interruption, and alerts on adaptive baselines and anomaly detection rather than fixed thresholds, delivering to Slack, email and webhooks. Cekura evaluates more than 60,000 voice AI calls daily on that surface.
Talk to Cekura about testing your ConversationRelay agent.
Frequently asked questions
Can you test a Twilio ConversationRelay agent before it is in production?
Yes. The agent answers on a Twilio number as soon as that number points at your TwiML endpoint, so a testing platform can dial it in staging. Cekura places inbound and outbound calls using imported Twilio or Plivo numbers, runs caller situations against the agent, and scores each turn, all before any real caller reaches the number.
What does Twilio's Conversation Relay Insights dashboard not cover?
It does not cover whether the caller's task succeeded, and it does not run before release. It reports time to first audio, interruption rate, silent calls and error codes for calls that already happened. Twilio also states it "is not a HIPAA Eligible Service", that its latency excludes last-mile latency to the end user, and that its measurements "are not guarantees of performance".
How do you test barge-in on a Twilio ConversationRelay agent?
Trigger the interruption deliberately and measure the overrun. Twilio's interrupt message carries utteranceUntilInterrupt and durationUntilInterruptMs, recording what the agent had said and how long synthesis continued past the caller's first word. Cekura measures interruption overrun in milliseconds independently of that message and derives mean, P50 and P90 across runs, so a regression in turn-taking shows up as a distribution shift.
Is Twilio's published latency low enough to skip testing the agent?
No. Twilio publishes median latency under 0.5 seconds and 95th-percentile latency under 0.725 seconds for ConversationRelay, described on its product page as based on internal benchmarks with results that may vary. That figure covers transport, speech recognition and synthesis. Your model call, your tool calls and your retrieval step sit on your WebSocket server and are not inside it.
Does Conversation Intelligence give you call audio for quality review?
No. The integration sends utterances, not audio, to Conversation Intelligence (classic), and it runs only after the call has ended. It also requires the intelligenceService attribute on the <ConversationRelay> noun, and its Language Operators will not run when the session language differs from the Intelligence Service language, which leaves the transcript stored but unscored.
How many times should each test scenario run?
More than once, and the same number of times every release. Cekura's benchmarks run 82 scenarios three times per configuration and rank on pass³, the share where all three retained runs passed, keeping calls that never connected in the denominator. Twilio ConversationRelay is not in that published cohort, so adopt the repetition method rather than the reported figures.
