Background noise robustness testing for voice agents

Background noise robustness testing for voice agents replays the same scripted calls under controlled noise types, signal-to-noise ratios (SNR) and phone codecs, then measures how transcription, turn-taking and task completion degrade. Cekura runs these tests with simulated callers whose personalities carry noise presets, custom audio files and network simulation, and scores every condition separately.

TL;DR

• Noise breaks turn-taking for some realtime models, not only transcription. On Cekura’s Turn Bench, Gemini 3.1 Flash Live answered 97% of finished caller turns in a quiet room and 55% in heavy noise (pink noise 10 dB below the voice), running on its provider’s default turn detection.
• Clean clips overstate real-call accuracy. On Cekura’s speech-to-text benchmark, 16 of 19 models had a higher word error rate (WER) on 8 real phone recordings than on 1,000 public clean clips. The real-call set is small and differs from clean clips in more than noise.
• Quiet-room rankings can flip in noise. In the Whisper paper, on LibriSpeech audio with noise added, several LibriSpeech-trained models beat Whisper at 40 dB SNR, yet all 14 comparison models did worse than Whisper in pub noise below 10 dB SNR.
• A useful noise test crosses noise type, SNR and channel (codec, echo, dropouts), then scores names and numbers, false barge-ins, endpointing and task completion against a clean run of the same script.
• Cekura sets background noise (built-in presets or any audio file by URL) and network simulation (packet loss, jitter, latency) on the caller’s personality, so one evaluator reruns under each condition and gets its own score.

What is background noise robustness testing for voice agents?

Background noise robustness testing is a voice agent test that holds the script, caller voice and expected outcome fixed and changes only the acoustic condition. The gap between the clean run and the noisy run is the cost of the noise, so you can rank fixes by it. A test that changes the script and the noise together cannot tell you which one caused a failure.

The conditions fall into two groups. Acoustic noise reaches the caller’s microphone: traffic, a TV, a coworker, a kitchen. Channel damage happens after the microphone: a narrowband phone codec, room echo, packet loss and dropouts. Real calls often carry both, so test each group alone first, then combine the worst pairs.

Cekura’s Turn Bench uses this design. Turn Bench plays 25 scripts to 13 realtime models, once clean and once through each of five degraded lines: an 8 kHz phone codec, light and heavy pink noise, a 0.4 s room echo, and 60 ms dropouts every 0.5 s.

How does background noise break a voice agent?

A noise-induced failure is a breakdown that starts in the audio and ends in a wrong reply or a wrong action. Background noise causes five of them, set out in the table. In a Twilio developer test of background noise, a running vacuum and drill made Deepgram Nova-3 on default settings hear “check-in window” as “second window”. ServiceNow’s EVA-Bench mixes noise into a simulated caller at a default 15 dB SNR across 12 voice agent systems. It found turn-taking the most perturbation-sensitive metric: 81% of measurements across accent and noise conditions degraded significantly.

“The model has no audio to fall back on.”

Dylan Frankcom, Software Engineer on Twilio’s Developer Content team, on the language model receiving only the transcript, in Twilio’s background noise tutorial (July 2026)

The last row is a single published example, not a failure rate, from a benchmark where providers chose their own models, speech components and settings. Check digits in the tool call, not only in the transcript.

What does benchmark data show about voice agents in background noise?

Cekura’s Turn Bench is a turn-taking benchmark that replays the same scripted calls to 13 realtime models under six line conditions. Its end-of-turn score, out of 100, asks two things: does the model wait through a caller’s pause, and does it reply once the caller finishes? Six models show the spread.

Each condition used 36 calls per model, streamed from US East with each provider’s default turn detection, so small gaps sit inside run-to-run variation and only the large drops count. Grok Voice Think Fast 2.0 loses 25 points between a quiet room and heavy noise, while GPT Realtime 2.1 loses 1.

Do speech-to-speech models still complete calls in background noise?

Cekura’s speech-to-speech benchmark is a test that runs full agent calls through each realtime voice model and scores the outcome, so it shows whether a call succeeds in noise rather than how well a turn is timed. Across 12 ranked models, background-noise calls averaged an 82% pass rate, from 62% (Gemini 3.8 Live) to 100% for four models: Grok Voice 3.0, GPT-Live 1 + GPT-6 Sol, Phonic v1 and Grok Voice Think Fast 2.0. That is mid-range among the benchmark’s ten kinds of call, which run from 41% (Medicare compliance and routing) to 94% (several requests in one call). Cekura’s speech-to-speech benchmark and Cekura’s Turn Bench rank these models differently. Both Grok models post the lowest heavy-noise end-of-turn scores in Turn Bench (60 and 57), yet passed every noisy call. GPT Realtime 2.1, Turn Bench’s pick for loud noise, passed 76%. Each model ran the same Pipecat agent, prompt and tools over live WebRTC calls, three runs per scenario.

Do clean speech-to-text benchmarks predict a voice agent’s accuracy in noise?

A clean-clip benchmark is a ranking built on short, well-recorded audio that says little on its own about accuracy on real phone calls. Cekura’s Converse-STT benchmark scores 19 models on 1,000 short public clips from Pipecat’s dataset and on 8 recordings from 4 real phone conversations licensed from Ocular. On the real calls, 16 of the 19 models made more errors. Zoom Scribe Pro had the lowest clean-clip WER at 1.75% (seven other models sit within its margin of error) and scored 4.83% on the real calls. GPT-4o Transcribe went from 3.90% on clean clips to 12.45% on real calls.

Read those numbers with two caveats. The real-call set is small, so its 95% ranges are wide and many models overlap. Real calls also differ from clean clips in more than noise: they are conversational and phone quality. The Whisper paper, which isolates noise, saw the same reordering below 10 dB SNR. Accents add a second axis, covered in Cekura’s guide to ASR accuracy testing for multilingual voice agents.

How do you build a background noise test for a voice agent?

A background noise test is a scenario suite run once clean and again under each noise condition. Build it in five steps:

1. Run a clean baseline several times. Cost: each repeat bills voice testing minutes.
2. Pick noise types by failure. Steady noise (road, fans, pink noise) lowers end-of-turn scores. Transients (doors, dogs) trigger false barge-in. Competing speech can silence an agent: Cekura’s speech-to-speech benchmark notes one model “can go silent under background conversation”. Cost: each type multiplies the matrix.
3. Sweep SNR at 20, 10 and 5 dB against active speech level as ITU-T P.56 measures it. That sweep is a recommendation, not a standard. Cost: three levels triple the runs.
4. Mix noise in first, then apply the phone codec, echo or packet loss, which degrade speech and noise together. Cost: each channel multiplies the matrix again.
5. Add real recordings from your callers’ environments, or an open corpus such as MUSAN: music, speech in 12 languages and many noises. Cost: someone must collect and label them.

What metrics measure a voice agent’s robustness to background noise?

A noise robustness metric is a score computed the same way on the clean run and the noisy run, so the gap between the two is the result. Report each one as noisy against clean for the same script:

1. Weighted transcription errors. Cekura’s Transcription Accuracy weights names, nouns, numbers and negation flips at 1.0, verbs at 0.5, and function words as minor variations.
2. False barge-ins. Cekura’s barge-in guide infers agent stops with no caller speech from noise-only calls, using interruption counts, overrun timing and silence metrics.
3. Endpointing. Track reply latency at p50 and p90, plus cut-offs where the agent spoke over the caller.
4. Turn coverage. Measure the share of finished caller turns the agent answered, which Turn Bench calls “replies when you’re done”.
5. Task outcome. Check that the spoken values reached the tool and the call ended in the expected state.

Raw WER covers only the first of the five, so a transcript-only suite passes agents that talk over callers or go quiet.

What counts as a pass in a background noise test?

A background noise pass is a scenario that reaches the same expected outcome as its clean baseline on every repeated run, not on average. Cekura’s speech-to-speech benchmark applies the same rule, called pass³: a scenario counts only when all three retained runs pass every required check, because some models pass a call on one run and fail the next. Set the tolerance from your own data. Run the clean baseline several times and treat a noisy score inside that run-to-run spread as no change, as Cekura’s Turn Bench does by reporting 95% intervals. Then score each condition as the clean result minus the noisy result on names and numbers transcribed, stops with no caller speech, reply timing, finished turns answered and task outcome. Cost: three runs per condition triple the voice testing minutes of a single pass.

How do you fix a voice agent that fails background noise tests?

A noise fix is a configuration change that trades one failure for a smaller one, so rerun the full matrix after each change. Each fix below has a cost:

1. Raise the barge-in threshold. Twilio’s Conversation Relay offers a low interruption sensitivity so only deliberate speech interrupts. Cost: callers must speak more deliberately to interrupt.
2. Ignore backchannels. Twilio’s ignoreBackchannel setting keeps “uh huh” from starting a new reply. Cost: a short real answer given mid-sentence can be ignored.
3. Tune end-of-turn confidence. Twilio’s tutorial lowers Deepgram Flux’s eotThreshold to 0.75 and warns against going below 0.7. Cost: lower values can cut off callers who pause mid-sentence.
4. Choose speech-to-text on your own noisy audio. Cost: on Cekura’s STT benchmark the model with the lowest overall WER returns final text in 235 ms and the fastest in 38 ms at the median, and many accuracy leads sit within margins of error.
5. Confirm critical values. Read digits back or offer keypad entry. Cost: one extra turn per value.

How does Cekura test voice agents for background noise robustness?

Cekura is a voice agent testing and monitoring platform that runs noise tests as simulated calls, with the condition set on the caller rather than the script. Cekura attaches background noise to the caller’s personality: built-in noise presets, or a Custom URL that plays any audio file, including recordings with real voices. Network Simulation on the same personality adds packet loss, jitter and latency. Evaluator instructions cannot override personality settings, so noise written into a script does nothing.

Cekura scores each run on Transcription Accuracy against the exact words the simulated caller spoke, plus interruption, latency and silence metrics. The Infrastructure Suite adds 18+ prebuilt scenarios, including audio quality under various conditions and packet loss, runnable in CI through GitHub Actions.

Two limits apply. The docs describe no per-run SNR setting, so record each noise file’s level and treat each file as one condition. Each condition also adds test minutes, at $0.25 per voice testing minute on pay as you go.

Frequently asked questions

What SNR should you test a voice agent at?

Test a voice agent clean and then at 20, 10 and 5 dB SNR, with noise measured against active speech level. That sweep is a recommendation, not a standard. Cekura’s Turn Bench uses pink noise 20 dB and 10 dB below the voice, EVA-Bench defaults to 15 dB, and the Whisper paper found models changed order below 10 dB SNR, so stopping at 20 dB can miss the failures that matter most.

Is synthetic noise enough to test a voice agent?

Synthetic noise is enough for regression testing and not enough for a launch decision. Pink noise is repeatable, so it shows whether a change made the agent worse. Real environments add speech, music and sudden transients that pink noise lacks. Run a synthetic sweep on every release, then add real recordings from your callers’ environments or an open corpus such as MUSAN before launch.

Does noise suppression fix background noise problems in voice agents?

Noise suppression fixes some background noise problems and creates others, so test it as its own condition. On Twilio’s Conversation Relay, your server never receives raw caller audio, which rules out common offline preprocessing libraries. Suppression stacked in two places can also misbehave, per the LiveKit guidance cited in Cekura’s echo cancellation guide. Run each scenario with suppression on and off, and keep the setting with better task completion.

Why does a voice agent stop talking when there is background noise?

A voice agent stops talking in noise because its barge-in detection reads a noise spike as the caller cutting in. In Twilio’s Conversation Relay tutorial, medium interruption sensitivity can stop the agent at each noise spike. Cekura’s Turn Bench shows how readily realtime models react to caller sounds: the typical model of 13 kept talking through only 6% of quick “mm-hmm” fillers. Raise the threshold, then confirm that real cut-ins still stop the agent fast.

Does Cekura test voice agents for background noise robustness?

Yes. Cekura runs background noise robustness tests as simulated calls, with noise presets or custom audio files and network simulation set on the caller’s personality. Cekura scores each run on transcription accuracy weighted toward names and numbers, plus interruptions, latency and silence. Transcription Accuracy also scores production call logs against two separate transcription models. To see it on your own agent, book a demo.