TL;DR
- Deepgram's Voice Agent API combines speech-to-text, LLM orchestration and text-to-speech over a single WebSocket at wss://agent.deepgram.com/, so there is no seam between components where you can insert a unit test.
- The API emits its own telemetry. Nineteen server messages include UserStartedSpeaking, AgentThinking, AgentStartedSpeaking and AgentAudioDone for turn timing, plus LatencyReport, which breaks each turn into stt_latency, four ttt_* think-stage fields and tts_latency, so a latency regression can be attributed to a stage without instrumenting your own bridge.
- Turn-taking is a configuration you own, not a default you inherit. Flux exposes eot_threshold (default 0.7), eager_eot_threshold and eot_timeout_ms (default 5000), and Deepgram's own documentation states that lowering the threshold buys "faster responses, more false positives".
- Orchestration choice moves reliability independently of the speech model. Per Cekura's voice orchestration benchmarks, across the four platforms that all ran Deepgram Nova-3 for speech-to-text, one byte-identical agent still ranged from 94.9% to 81.4% on pass^3 and from 2.34s to 3.16s on p50 turn latency, on platform defaults rather than tuned configurations. Deepgram Voice Agent is not one of the platforms in that dataset.
- Production monitoring for a Deepgram agent is a push, not a pull. Because you host the WebSocket client, no vendor dashboard sees your calls, so transcripts have to be forwarded to an evaluation platform after each call ends.
What makes testing a Deepgram Voice Agent different?
The Deepgram Voice Agent API is a single-connection agent runtime that, per Deepgram's Voice Agent documentation, "handles the full speech pipeline (listening, thinking, and speaking)". Speech-to-text, LLM orchestration and text-to-speech run behind one WebSocket connection rather than three services you wire together.
That architecture is the reason the usual test strategy fails. When you assemble a pipeline yourself, you can test STT accuracy on a fixture set, prompt logic in text, and TTS output in isolation, because each boundary is a function call you control. Deepgram's runtime removes those boundaries. The agent is only observable as a conversation.
Dashed markers show where a unit test can attach. Deepgram's runtime has none.
Two consequences follow. First, correctness and timing become the same test: the same connection that decides what to say also decides when to stop listening. Second, your test harness has to speak. A text-mode eval can still check prompt logic, but it cannot reach end-of-turn detection, barge-in, or the telephony bridge, which is where most production failures live.
Which tools test voice AI agents built on Deepgram Voice Agent?
The tools that test voice AI agents built on Deepgram Voice Agent fall into three groups, and each group reaches a different depth of the stack. Deepgram's own surfaces cover component behavior. Your harness covers the assembled agent. An evaluation platform such as Cekura covers the assembled agent at scale, with scoring and regression history attached.
| Layer | What Deepgram provides | The failure it produces | Signal that exposes it |
|---|---|---|---|
| Listen (Nova-3, Flux) | Streaming STT with model-integrated end-of-turn detection | Misheard names, numbers and spelled-out entities under accent or noise | ConversationText compared against the scripted caller utterance |
| Turn-taking (Flux) | eot_threshold, eager_eot_threshold, eot_timeout_ms | Agent cuts the caller off, or leaves multi-second dead air | Gap between UserStartedSpeaking and AgentStartedSpeaking |
| Think (LLM) | Configurable LLM provider and prompt, updatable mid-session | Wrong tool, skipped required field, invented policy | Transcript scored against the scenario's expected outcome |
| Function calling | FunctionCallRequest and FunctionCallResponse | Correct intent, malformed arguments, or a silent hang while waiting | Argument-level assertions on the emitted request |
| Speak (Aura-2) | TTS with sub-200ms time-to-first-byte | Mispronounced domain terms, truncated audio on interruption | AgentAudioDone versus the audio actually delivered to the caller |
| Session lifecycle | Two-hour cap, MAXIMUM_SESSION_LENGTH_APPROACHING warning five minutes ahead | Long hold or transfer call dies mid-conversation | Warning and Error server messages |
| Telephony bridge | Not Deepgram's layer. Yours, via Twilio or SIP | Packet loss, one-way audio, DTMF ignored | A real inbound call, not a WebSocket test client |
The rightmost column is the practical point: every row is measurable from the event stream Deepgram already emits, so the work is capturing those events per call and scoring them, not building new instrumentation.
The Listen and Function calling rows also compound, which is why they are scored together rather than separately. A misheard entity that still yields a fluent transcript, a caller's "fifty" logged as "fifteen", reads as a correct turn to any judge looking only at the transcript, and surfaces only as a wrong argument in FunctionCallRequest. A clean transcript is not evidence of a clean tool call.
Deepgram's Voice Agent feature overview documents the full event and message set, and the WebSocket API reference lists all nineteen server messages and ten client messages by name.
How to test a Deepgram Voice Agent voice agent
You test a Deepgram Voice Agent voice agent by driving it as a caller, not as an API. Cekura connects to a Deepgram agent through the entry point the agent already exposes, because Deepgram Voice Agent is a runtime you host rather than a managed platform with a vendor-side test hook.
Three connection paths work today. If the agent is fronted by a phone number, Cekura places inbound and outbound calls using imported Twilio or Plivo numbers. If it terminates SIP, Cekura sends a SIP INVITE to the agent's SIP URI and runs the scenario over the established media path. If neither applies, your bridge forwards each finished call to Cekura's webhook and the transcript is scored there.
From there the loop is ordinary regression discipline, covered in general terms in our guide to testing voice AI agents and in our automated QA best practices for voice agents. Fix a scenario suite, run every scenario more than once, and gate on the pass rate rather than a single green run. Cekura's published suites include 50+ caller personalities and 25+ predefined metrics, plus an Infrastructure Suite that "includes 18+ test scenarios covering hold tests, audio quality under various conditions, interruption handling, language support, and edge cases like packet loss and rapid-fire phrases".
Tune and test turn-taking as one step
Flux is the layer most worth an explicit test pass, because its defaults are a choice about your callers. Deepgram describes Flux as offering "first-of-its-kind model-integrated end-of-turn detection, configurable turn-taking dynamics, and ultra-low latency optimized for voice agent pipelines, all with Nova-3 level accuracy".
The tradeoff is stated plainly in the vendor's own reference, which is why it belongs in a test plan rather than a tuning session:
"Higher values (e.g., 0.8 - 0.9) = Higher certainty required before ending a turn, fewer false positives, slightly increased latency" versus "Lower values (e.g., 0.5 - 0.7) = Lower certainty required before ending a turn, faster responses, more false positives"
— Deepgram, Flux configuration reference
Deepgram weights this the same way in its own conversational benchmark. The Voice Agent Quality Index combines Interruptions at 40% weight, Missed Responses at 40% and Latency at 20%, so on the vendor's own scoring four fifths of conversational quality is turn-taking behavior and one fifth is speed. Cekura scores both of those behaviors per scenario, which is why the interrupter and pauser personalities run on every release rather than only during a latency investigation.
Test both directions of that tradeoff with caller behavior, not with settings alone. A caller who pauses mid-sentence to read a card number exposes an eot_threshold set too low; a caller who finishes crisply exposes an eot_timeout_ms of 5000 as five seconds of dead air. The interrupter and pauser personalities hold each end of that range. Our guide to endpointing and turn detection covers the underlying signals in more depth.
How to monitor Deepgram Voice Agent voice agents in production
Monitoring a Deepgram Voice Agent in production means forwarding every completed call to an evaluation platform, because nothing else can see your traffic. You host the WebSocket client, so there is no vendor-side console holding your call history the way a managed platform would. Cekura ingests those transcripts, scores each call against the same metrics used in pre-release testing, and alerts on the metrics that move.
The mechanism is a webhook. Per Cekura's custom integration documentation, your backend sends call data after each call ends, authenticated with an X-CEKURA-API-KEY header, and "Cekura listens for call data for 5 minutes after the call ends." Each call object requires an id, a startedAt and an endedAt in ISO 8601. Because the same evaluators run on production calls as on simulated ones, a regression caught in production is directly comparable to the pre-release run that missed it.
What to monitor, and from which event
The Deepgram event stream carries most of what production monitoring needs. The value of forwarding it is that per-call fields become distributions you can alert on.
| Signal | Source | What it catches | Alert on |
|---|---|---|---|
| Turn latency p50 and p90 | LatencyReport.total_latency, or the gap from UserStartedSpeaking to AgentStartedSpeaking | Think-time creep after a prompt or model change | p90 drift, not the mean |
| Latency attribution by component | LatencyReport fields stt_latency, ttt_token_latency, ttt_text_latency, ttt_tool_latency, ttt_thinking_latency and tts_latency | Which stage of STT, LLM or TTS absorbed a regression | The component whose p90 moved, not the total |
| Barge-in rate | UserStartedSpeaking received between AgentStartedSpeaking and AgentAudioDone | Agent too slow or too verbose, so callers talk over it | Sustained rise versus the pre-release baseline |
| Premature end-of-turn | Caller turns that resume immediately after the agent starts | eot_threshold set too aggressively for real callers | Rate per 100 calls |
| Dead air | Long gaps with no AgentThinking following caller speech | Upstream LLM or tool stall | Any gap beyond your conversational budget |
| Tool-call correctness | FunctionCallRequest arguments | Right intent, wrong arguments | Argument-level failure rate |
| Session deaths | MAXIMUM_SESSION_LENGTH_APPROACHING, Error | Long calls hitting the two-hour cap | Any occurrence |
| Task completion | Transcript scored by an LLM judge | The failure class no latency metric shows | Completion rate by scenario type |
Deepgram breaks that budget down for you rather than making you infer it. Per Deepgram's Voice Agent observability documentation, the LatencyReport message carries stt_latency, ttt_token_latency, ttt_text_latency, ttt_tool_latency, ttt_thinking_latency, tts_latency and total_latency, all floats in seconds, each optional and omitted when it does not apply to that turn. Log every field per turn. A total that rises while stt_latency and tts_latency hold flat localizes the regression to the think stage, which is the stage your prompt and model changes actually touch.
Turn latency is the gap from UserStartedSpeaking to AgentStartedSpeaking. LatencyReport reports the same interval directly and breaks it into stt_latency, the ttt_* think-stage fields and tts_latency.
Set the latency budget against a stated conversational threshold rather than a round number. Deepgram's own position is that "in human conversation, delays beyond 1000ms feel unnatural", and Aura-2 contributes sub-200ms time-to-first-byte to that budget, which leaves think time as the variable you actually control.
Deepgram documents the transcription half of that budget directly. Per Deepgram's streaming latency guide, its models are "optimized to deliver transcription latency in 300 milliseconds or less", and the page puts typical total transcript latency at 200 to 500 ms and Flux end-of-turn detection at 100 to 500 ms. Measure transcript latency the way that page prescribes rather than by wall clock: track the seconds of audio you have sent as X, the seconds of audio represented in the latest interim transcript as Y, and record X minus Y per turn. Deepgram states that transcript latency and end-of-turn latency are different metrics for different use cases and instructs you to measure them independently, so alert on them as two separate p90s rather than folding both into a single turn-latency number.
Two operational cautions. Alert on p90, because a p50 that holds while p90 stretches is the signature of a change that only hurt the harder calls. And redact before you store: production call transcripts carry caller data, so PII redaction belongs in the ingestion path, not in a later cleanup. For a broader comparison of approaches to production monitoring across platforms, see our guide to voice agent monitoring platforms.
Why can't you infer reliability from Deepgram's model accuracy?
Component accuracy does not predict assembled-agent reliability, and there is direct evidence for the size of the gap. Deepgram Nova-3 measures 5.2% word error rate on Artificial Analysis's speech-to-text leaderboard at $4.30 per 1,000 minutes of audio, under the AA-WER v2 index, checked August 2026. That is a third-party index figure, not a vendor-reported one, and the two diverge often enough that it is worth stating which you are quoting. Word error rate on a leaderboard audio set is not task completion on your calls, which is the gap the rest of this section measures. Per Cekura's voice orchestration benchmarks, one agent was deployed unchanged across six orchestration platforms with GPT-4.1 at temperature 0 and a byte-identical prompt, then called over the phone by the same 59 evaluators, each scenario run three times.
The speech layer could be pinned on only four of the six. Nova-3 was pinned on Vapi, Synthflow, LiveKit and Pipecat, while Retell exposes only a coarse STT mode and ElevenLabs forces its own Scribe model. That makes those four a controlled comparison: same speech model, same LLM, same prompt, different orchestration. Across them, pass^3, the share of scenarios succeeding on all three runs, still ranged from 94.9% down to 81.4%, and median per-turn latency ranged from 2.34s to 3.16s. Across all six the spread is wider, 96.6% to 76.3% and 1.73s to 3.16s, but two of those platforms ran different speech models so that wider range is not a clean orchestration comparison. Every figure is a platform default, and the benchmark states plainly: "these are platform defaults, expect higher peaks once you tune to your use case."
Deepgram Voice Agent is not one of the platforms measured in that dataset, so none of those numbers describe it. What transfers is the controlled finding: with the speech model held fixed, the orchestration layer alone moved pass^3 by more than thirteen points. Nova-3's word error rate tells you nothing about whether your agent completes its task three times out of three, which is why the assembled agent has to be measured directly. Our voice AI evaluation metrics guide covers which measurements are worth gating on.
What should you test beyond the happy path?
A passing scenario suite covers the calls you imagined. Three classes of test cover the calls you did not.
Load. Concurrency changes latency, and latency changes conversational behavior. Deepgram's flat-rate pricing of $4.50/hr makes sustained concurrent testing affordable to run, so measure p90 at your expected peak rather than at one call. Our guide to voice load testing covers the method.
Adversarial. The UpdatePrompt and InjectAgentMessage client messages give you mid-session control, and anything that can steer an agent mid-session is worth probing from the caller side too. Cekura's published red-team library covers 10,000+ scenarios for prompt injection and boundary testing, and our penetration testing guide covers scope.
Recovery. Test the failure paths Deepgram documents rather than assuming they will not fire: the two-hour session cap, the five-minute advance warning, and the Error and Warning messages. A long hold or a warm transfer is exactly the call that reaches them.
Acoustic. Word error rate on a leaderboard audio set is measured on clean audio, and your callers are not on clean audio. Run the same scenario suite over degraded conditions rather than assuming the speech layer holds: background noise, packet loss and non-native accents. Cekura's Infrastructure Suite covers audio quality and packet loss as first-class scenarios. Score ConversationText against the scripted utterance per condition, so accuracy loss is attributed to the condition rather than to the agent.
Frequently asked questions
How do you monitor Deepgram Voice Agent voice agents in production?
Forward each completed call to an evaluation platform, because you host the WebSocket client and no vendor console holds your call history. Cekura accepts call data by webhook after each call ends, authenticated with an API key header, then scores production transcripts on the same metrics used pre-release. Alert on p90 turn latency, barge-in rate and task completion rather than averages.
How do you test a Deepgram Voice Agent voice agent?
Drive it as a caller. Cekura places calls through the agent's existing entry point, either an imported Twilio or Plivo phone number, a SIP INVITE to its SIP URI, or a webhook that forwards finished calls for scoring. Run a fixed scenario suite more than once per release and gate on the repeated pass rate, not a single successful run.
Does Cekura's orchestration benchmark include Deepgram Voice Agent?
No. Cekura's benchmarks cover Retell, Vapi, Pipecat, LiveKit, Synthflow and ElevenLabs, all on platform defaults rather than tuned configurations. Deepgram Nova-3 was pinned on four of them, Vapi, Synthflow, LiveKit and Pipecat, so those four form a controlled comparison, and pass^3 across them still spanned 94.9% to 81.4%. That is what shows orchestration moving reliability independently of the speech model. The benchmark provides no numbers for Deepgram Voice Agent.
Does Cekura have a native Deepgram Voice Agent integration?
Cekura's named platform integrations are Retell, Vapi, ElevenLabs, LiveKit and Pipecat. Deepgram Voice Agent is reached instead through the generic paths, which cover it fully: telephony with your own Twilio or Plivo numbers, SIP, or the custom webhook integration for transcript ingestion. No Deepgram-specific setup is required.
Can you test a Deepgram Voice Agent without a phone number?
Yes. If the agent is not fronted by telephony, your bridge can forward each completed conversation to Cekura's webhook for scoring, and text-mode simulation can exercise prompt logic before any audio is involved. Both skip the telephony path, so neither substitutes for at least one real call per release covering packet loss, DTMF and one-way audio.
What latency should a Deepgram Voice Agent hit?
Deepgram states that "in human conversation, delays beyond 1000ms feel unnatural" and that Aura-2 delivers sub-200ms time-to-first-byte, which makes think time the variable you control. Set a p90 budget rather than a p50 target, measure it from UserStartedSpeaking to AgentStartedSpeaking, and re-measure after every prompt or model change.
Which Deepgram events matter most for testing?
Five carry most of the signal: ConversationText for what was heard and said, UserStartedSpeaking and AgentStartedSpeaking for turn timing and barge-in, AgentThinking for stalls, and FunctionCallRequest for tool-call arguments. LatencyReport reports timing directly and attributes it per stage through stt_latency, the ttt_* think-stage fields and tts_latency, and Warning and Error surface session-level failures including the two-hour cap.
Should you trust Deepgram's published word error rate?
Quote the third-party figure and say so. Artificial Analysis lists Deepgram Nova-3 at 5.2% word error rate under its AA-WER v2 index, checked August 2026, at $4.30 per 1,000 minutes of audio. Vendor-published word error rates are measured on vendor-chosen audio, so they answer a different question. Neither figure predicts task completion on your calls, which is why Cekura scores the assembled agent rather than the speech layer.
Testing a Deepgram Voice Agent with Cekura
Cekura tests and monitors voice agents built on the Deepgram Voice Agent API through the entry point the agent already exposes: an imported phone number, a SIP URI, or a webhook that forwards completed calls. The same evaluators run pre-release and in production, so a regression found on a live call is comparable to the run that missed it.
Book a demo to run a scenario suite against your Deepgram agent.