Voice AI agents built on Twilio ConversationRelay are tested by dialling the agent's Twilio number, scoring each conversation against per-layer evaluators, and gating the release on how often every run passes. Cekura places those calls using imported Twilio numbers, evaluates the transcripts, and monitors the same agent in production.
TL;DR
- ConversationRelay runs half your agent. Twilio owns speech recognition, speech synthesis and the media path; your WebSocket server owns the model, the retrieval step and the tool calls. Neither half logs the other.
- Only two of the five options are automated software tools that return a repeatable pass rate: tests against your own WebSocket server, and a call-placing platform such as Cekura. Twilio's two observability products describe calls that already happened.
- Twilio publishes a set of ConversationRelay error codes, and two of them are silent in application logs: a session closed after ten consecutive unidentified messages, and a dropped WebSocket that Twilio does not reconnect.
- Twilio states median latency under 0.5 seconds and 95th-percentile latency under 0.725 seconds for Conversation Relay, footnoted on its own product page as based on internal benchmarks with results that may vary. It covers Twilio's half only.
- End-to-end quality assurance means dialling the number a customer would dial, then running the same caller situation again after every prompt or model change.
Why do agents built on Twilio ConversationRelay need their own testing tools?
They need their own tools because ConversationRelay splits one agent across two systems, and neither system holds the whole call.
The <ConversationRelay> noun sits inside <Connect> and opens a WebSocket to a server you wrote. Twilio runs speech recognition, speech synthesis and the media path. Your server runs the model, the retrieval step and the tool calls. A wrong booking can originate on either side, and each side logs only its own half.
Twilio's WebSocket message reference sets the contract. Twilio sends setup on connect, prompt when the caller speaks, dtmf on keypresses, interrupt when the caller talks over playback, and error on session faults. Your server replies with text tokens, play media, send digits, language switch and end session messages.
Two failure modes fall directly out of that contract, and neither one appears in your application logs as a defect. Twilio terminates the session when the last ten messages from your WebSocket remain unidentified, closing with status code 1007 and the reason "Too many consecutive malformed messages". And when the WebSocket drops, Twilio documents that it does not reconnect: the call disconnects with a failed status. Only a real call surfaces either one.
What are the best automated software tools for testing voice AI agents built on Twilio ConversationRelay?
Five options cover a ConversationRelay agent, and only two of them are automated software tools in the sense that matters, meaning they run without a person on the call and return a pass rate you can gate on.
| Tool | Automated | Which half of the call it reaches | When it runs |
|---|---|---|---|
| Manual test calls over a tunnel | No | Both, but only on the path you happened to try | Ad hoc, before release |
| Unit and integration tests on your WebSocket server | Yes | Your half: TwiML generation, message handling, tool-call code | In CI, on every commit |
| Conversation Relay Insights | No, it is a reporting dashboard | Twilio's half: timing, interruptions, error codes | After calls run in production |
| Conversation Intelligence (classic) | Partly, operators run automatically | Transcript only, and only once the call has ended | Post-call, when intelligenceService is set |
| Cekura | Yes | Both halves, from the caller's audio to the task outcome | Before release and in production |
The dividing line is direction, not depth. Twilio's two observability products describe calls that already happened, which is the right shape for an incident review and the wrong shape for a release gate. Server-side tests run early and cheaply, but they exercise your code rather than audio, so they will not catch a digit misheard through a codec or an agent that speaks straight through an interruption.
That leaves one gap: a tool that dials the agent before the change ships. Cekura fills it, and pairs with server tests rather than replacing them. Wire both into the same pipeline the way Cekura describes for GitHub Actions.
How do you perform end-to-end quality assurance for voice AI agents integrated with Twilio ConversationRelay?
You perform end-to-end quality assurance on a ConversationRelay agent by dialling it exactly as a customer would, because a phone number is already the agent's front door.
The number you pointed at your TwiML endpoint is what makes the agent answerable, so a testing platform needs nothing more than that number. Cekura states the option directly: "If the agent is fronted by a phone number, Cekura places inbound and outbound calls using imported Twilio or Plivo numbers." Cekura documents the SIP and webhook alternatives for agents without a number in Tools to Test Voice AI Agents Built on the Deepgram Voice Agent API.
Five steps take an agent from reachable to release-gated:
- Point the Twilio number at the TwiML endpoint returning your
<Connect><ConversationRelay>response. - Import that number so scenarios dial the real agent rather than a mock.
- Write scenarios as caller situations rather than fixed scripts.
- Attach evaluators per layer: transcription, turn-taking, tool-call arguments, task outcome, voice quality.
- Run every scenario more than once, and gate the release on how often all runs pass.
Step 5 is the one teams skip, and the one that decides whether your pass rate means anything.
Which are the top-rated platforms for monitoring and debugging voice AI agents that utilize Twilio ConversationRelay?
Twilio ships two products, and they split the job: one for timing, one for what was said. Debugging means deciding which half of the WebSocket failed, then reading the transcript to see what it cost the caller.
Conversation Relay Insights reports average time to first audio, calls where the agent took longer than 1.2 seconds to respond, interruption rate, silent calls, words per token and the top five error codes. It also times the gap between the last speech-to-text block sent to your WebSocket and the first token you send back, the one number that indicts your half. Twilio states three limits: it "is not a HIPAA Eligible Service", its speech-to-text latency is typically accurate within 100ms for English and up to 250ms elsewhere, and these measurements "are not guarantees of performance".
Conversation Intelligence (classic) adds transcript persistence and Language Operators for task completion, escalation, hallucination and toxicity. Both are read-only views of production traffic, so neither tells you whether the change in your branch breaks a booked flow.
How do you debug a ConversationRelay call that ended badly?
Start from Twilio's error code, because it tells you which side to open first, then reproduce the call rather than reasoning about it.
| Error code | What it means | Ends the session |
|---|---|---|
| 64101 | Invalid Parameter, meaning invalid TwiML configuration parameters | Yes |
| 64102 | Unable to Connect to WebSocket URL | Yes |
| 64105 | WebSocket Ended, your connection disconnected unexpectedly | Yes |
| 64107 | Invalid Message Received; the connection closes after 10 consecutive invalid messages | No |
| 64108 | RTP Timeout, no audio packets received | Yes |
| 64109 | Concurrency Limit Reached for the account | Yes |
Codes 64101, 64102 and 64109 are configuration and capacity problems, and a single staging call reproduces every one of them in seconds, before a customer ever meets one. 64105, 64107 and 64108 are the hard ones, because the agent looked healthy on your side until it stopped.
Turn-taking bugs leave no error code at all. Twilio's interrupt message carries utteranceUntilInterrupt and durationUntilInterruptMs, recording what the agent had already said and how long synthesis continued past the caller's first word. Most teams discard that message. Cekura measures the same overrun independently, capturing how long TTS continued after user speech began, so a regression appears as a shift in the distribution rather than a complaint.
Why should each test scenario run more than once before you ship?
Because one green call proves the agent can succeed, not that it will, and the difference between those two is where production incidents live.
Cekura's benchmark runs 82 scenarios three times against each of seven platform configurations and ranks on pass³, the share of scenarios where all three retained runs passed. Calls that did not connect or produced no transcript stay in the denominator instead of being dropped. Twilio ConversationRelay is not one of the seven configurations, so the method transfers and none of the figures do. Two caveats travel with any number from that study: providers chose the configurations they submitted, the exception being OpenAI, which Cekura tested directly on gpt-realtime-2.1 with no configuration submitted by OpenAI; and response time there is Cekura's main-agent measure, not provider-native component timing.
The cohort shows the trap in one row. Vapi records the highest task completion rate in the set at 97.56% across calls with outcome evidence, and still ranks sixth of seven on repeatable reliability at 59.76% pass³, because 41 of its 246 retained calls never connected at all. A team reading only task completion would have shipped that agent.
How does Cekura test and monitor a ConversationRelay agent?
Cekura runs the same evaluator set before release and in production, so a regression caught in staging and one caught live are the same measurement rather than two different reports.
Before release, Cekura dials the agent's Twilio number, runs caller situations that vary wording, accent, interruption timing and background noise, scores each turn against per-layer evaluators, and reports which scenarios stopped passing after a prompt or model change. That covers both halves of the WebSocket, because the call enters through the same audio path a customer uses.
In production, Cekura scores live traffic against that evaluator set, tracking hallucination, relevancy, latency, tool-call success, interruption and overtalk patterns, and stop time after an interruption. Alerts fire on adaptive baselines and anomaly detection rather than fixed thresholds, and deliver to Slack, email and webhooks.
If you are still choosing which layer of the stack to build on rather than how to test it, Cekura compares the options in Twilio vs Vapi. If the agent is already answering calls, talk to Cekura about testing it.
Frequently asked questions
Can you test a Twilio ConversationRelay agent before it reaches production?
Yes. The agent answers as soon as a Twilio number points at your TwiML endpoint, so a testing platform can dial it in staging. Cekura places inbound and outbound calls using imported Twilio or Plivo numbers, runs caller situations against the agent and scores every turn, all before a real caller reaches the number.
What does Conversation Relay Insights not tell you?
It does not tell you whether the caller's task succeeded, and it does not run before release. It reports time to first audio, calls above the 1.2 second threshold, interruption rate, silent calls, words per token and top error codes for calls that already happened. Twilio also states it "is not a HIPAA Eligible Service" and that its measurements "are not guarantees of performance".
Does Conversation Intelligence give you the call audio for quality review?
No. Twilio documents that Conversation Relay sends the transcribed utterances but not the audio, and that the audio is not available in the Transcripts Viewer or the API. Transcripts appear only after the call ends or after you send the session end message. If PCI mode is enabled on the account, the transcript is rejected and no Language Operators run.
Why would Language Operators store a transcript without scoring it?
Because the languages do not match. Twilio states that when the language set on the Conversation Relay session differs from the language set on the Intelligence Service, the transcript is stored but Language Operators will not run. Switching language mid-session is the related case: operators attempt to run, but Twilio warns results may be inaccurate for multi-language transcripts.
Is Twilio's published latency low enough to skip testing the agent?
No. Twilio publishes median latency under 0.5 seconds and 95th-percentile latency under 0.725 seconds for Conversation Relay, footnoted as based on internal benchmarks with results that may vary. That range covers transport, speech recognition and synthesis. Your model call, your retrieval step and your tool calls run on your WebSocket server and sit outside the measurement entirely.
How many times should each scenario run?
More than once, and the same number of times every release so the figures stay comparable. Cekura's benchmark runs 82 scenarios three times per configuration and ranks on pass³, the share where all three retained runs passed, keeping calls that never connected in the denominator. ConversationRelay is not in that cohort, so adopt the repetition method rather than the reported numbers.
