Voice activity detection is the classifier that decides, dozens of times per second, whether your caller is speaking or silent. You tuned the prompt and swapped the model, but the component deciding whether your agent talks over a caller is a two-megabyte file you probably never configured.
What Is Voice Activity Detection? The 30-Second Answer
Voice activity detection (VAD) is a binary classifier that labels each short window of audio as speech or non-speech. Inside a voice agent, its job is to tell the rest of the stack when to listen, when to wait, and when it is safe to respond.
Bottom line: VAD is the first decision in your pipeline, and every component downstream inherits its mistakes.
Key Features
Frame-level decisions. The detector scores tiny slices of audio rather than whole utterances. The G.729 speech coding standard makes an active or inactive call every 10ms frame.
Probability output. Modern detectors emit a float between 0 and 1, and you pick the cutoff.
Language independence. Silero VAD was trained on corpora spanning over 6,000 languages, so it generalizes across accents and domains.
A tiny compute footprint. Silero processes a 30ms chunk in under 1ms on a single CPU thread, and its JIT model is around two megabytes.
| Detector | 🧠 Signal it uses | 📦 Deployment | ✅ Best for |
|---|---|---|---|
| Silero VAD | Acoustic, neural | Self-hosted, PyTorch or ONNX, MIT | Phone agents you host yourself |
| OpenAI server_vad | Duration of acoustic silence | Managed inside a Realtime session | Agents already on the Realtime API |
| OpenAI semantic_vad | The words spoken so far | Managed inside a Realtime session | Callers who pause to think |
| pyannote/segmentation-3.0 | Acoustic, neural segmentation | Self-hosted via Hugging Face | Offline diarization and call analysis |
| pipecat-ai/smart-turn-v3 | Turn completion | Self-hosted via Hugging Face | Open-source stacks adding semantic turn-taking |
Hugging Face currently lists around 200 models tagged for voice activity detection, with pyannote's segmentation model pulling more than 6 million downloads a month.
How Does Voice Activity Detection Work?
A VAD runs the same loop continuously against your audio stream. The typical design applies noise reduction, extracts features from a frame, and then applies a classification rule to that frame.
In practice, the pipeline has five stages:
- Capture: Audio arrives from telephony or WebRTC.
- Framing: The stream is sliced into windows. Silero's default window is 512 samples, which is 32ms at 16kHz.
- Feature extraction: Raw samples become numbers a classifier can use.
- Classification: The frame is scored.
- Probability output: A value between 0 and 1 goes to whatever logic you wrote.
Real example: a caller says "It's for, uh, the eighth." The detector emits high probabilities across "It's for," drops during the pause, spikes again on "uh," drops once more, then spikes on "the eighth."
That's four speech regions where a human hears one sentence, and this is the source of a lot of the challenges with voice agents.
Signal-Processing VAD
The oldest detectors threshold on energy. Loud frame equals speech, quiet frame equals silence, with zero-crossing rate and spectral shape sometimes layered on.
The approach is cheap, and it fails in the exact conditions phone agents live in. As the BT Labs and University of Liverpool engineers who built a voice activity detector for G.729 put it, energy thresholding is inadequate where background noise is high, because loud non-speech gets misclassified as speech.
That paper is from 1997, but the failure mode has not aged.
Neural VAD
Neural detectors learn the speech-versus-noise boundary from data instead of from hand-written rules. Silero VAD is the one most open-source voice stacks reach for, and it is MIT licensed with no telemetry, keys, or registration.
It supports 8kHz and 16kHz sampling rates, which matters more than it sounds. Your lab recordings are 16kHz. Your phone calls are not.
Semantic VAD
Acoustic silence is ambiguous. A breath and a finished thought look identical to a detector listening only to sound.
Semantic detectors score the words. OpenAI's semantic_vad mode runs a classifier that estimates the probability the caller is done, then adjusts its timeout accordingly.
Audio trailing off with an "ummm" earns a longer wait than a definitive statement. You tune it with an eagerness setting of low, medium, high, or auto.
VAD vs. Endpointing: What's the Difference?
These get used interchangeably, and they are different layers.
| Voice activity detection | Endpointing | |
|---|---|---|
| Question it answers | Is there speech in this frame? | Has the caller finished their turn? |
| Scope | One window of audio | The whole conversational handoff |
| Inputs | Acoustic signal | VAD, transcript finality, semantics |
| Output | A probability | A decision to respond |
There is no winner here because they are not competing. VAD is an input, and endpointing is the decision that consumes it. A detector with excellent frame accuracy still produces a terrible agent if the logic above it responds after 80ms of silence.
For how that decision layer combines acoustic, lexical, and semantic signals, our guide to endpointing and turn detection covers the full picture.
What VAD Gets Right and Where It Falls Short
Pros (What Actually Works)
✅ It costs almost nothing to run. Sub-millisecond inference on one CPU thread means you can run it on every call with no capacity planning.
✅ It works across languages you never tested. A detector trained on thousands of languages will handle a caller you did not plan for.
✅ It saves real money downstream. Gating your speech-to-text on voiced frames means you stop paying to transcribe silence.
✅ It runs anywhere. ONNX and PyTorch runtimes cover browsers, phones, servers, and single-board computers.
Cons (Where It Falls Short)
Every detector makes four kinds of errors. The G.729 researchers named them in 1997, and the taxonomy still maps cleanly onto voice agent failures today.
Front-End Clipping (FEC)
❌ What the caller hears: they say "eight," and the agent books the 18th.
❌ What causes it: frames at the start of speech get classified as silence, so the opening syllable never reaches your speech-to-text. The 1997 paper attributes this to low-amplitude speech swamped by background noise.
❌ How you catch it: compare transcripts against known scripted input. Consistent loss of leading consonants and digits points at FEC. Audit the detector before the model.
Mid-Speech Clipping (MSC)
❌ What the caller hears: the agent interrupts them mid-sentence.
❌ What causes it: speech partway through an utterance gets misclassified as silence, usually during a low-amplitude passage. Your endpointing logic sees silence and hands the turn over.
❌ How you catch it: stereo recording. If the agent channel goes active while the caller channel still has speech, you have an overlap event.
Hang Over (HO)
❌ What the caller hears: dead air after they stop talking.
❌ What causes it: this one is deliberate. Detectors delay the silence decision, so low-amplitude mid-speech is not clipped. The G.729 work used fifteen 10ms frames of overhang, against five in the GSM detector.
❌ How you catch it: track response latency at p50, p95, and p99. An average of 400ms hides a p99 of 2.5 seconds.
Noise Detected as Speech (NDS)
❌ What the caller hears: the agent responds to a cough, a car horn, or a coworker.
❌ What causes it: background noise inside a silence period gets classified as speech.
❌ How you catch it: run your suite with noise layered over the caller audio and count the turns the agent opened without a real utterance.
The tradeoff: these errors pull against each other. Suppress the hangover, and you gain mid-speech clipping. But if you raise the threshold to kill noise-detected-as-speech, you gain front-end clipping. There is no setting that zeroes all four.
One important distinction is that NDS and HO measure efficiency, while FEC and MSC map to what the caller actually perceives. The authors found that any FEC+MSC total below 1.0% produced speech with no perceptual clipping distortion. That is a real target you can hold your detector to.
Should You Rely on VAD Alone? Our Take
Acoustic detection alone is enough for some jobs and thin for the one most people are building.
The Cekura voice orchestration benchmark ran one byte-identical agent across six platforms, with 59 evaluators and three runs each.
Latency, interruption handling, repetition, and end-call behavior tracked the orchestration layer, which means voice activity detection and end-of-turn detection, rather than the model. It was the same prompt, same LLM, and same voice, but materially different conversations.
The interruption spread was tight and directional. Pipecat scored 4.90 out of 5, LiveKit 4.89, and Vapi 4.63 on scenarios injecting barge-in, coughs, and mid-turn silence. Median per-turn latency ranged from 1.73s to 3.16s across the total six platforms.
VAD Alone Is Enough If You:
- Gate speech-to-text to cut transcription spend
- Trim silence out of recordings before storage
- Reduce bandwidth by not transmitting silent packets
- Segment audio for offline diarization
Add a Second Signal If You:
- Ship a phone agent that talks to strangers
- Handle numbers, spellings, or addresses, where clipping changes meaning
- Serve callers who pause to find their insurance card
- Run in any environment noisier than an office
How to Configure Voice Activity Detection in 5 Steps
1. Pick the detector for your runtime: Self-hosting on Pipecat or LiveKit points at Silero, already on the Realtime API points at server_vad or semantic_vad.
2. Set your threshold: Probabilities above this count as speech. Silero ships 0.5, and the maintainers are blunt in their own source that "lazy" 0.5 is decent for most datasets and better tuned per dataset. A higher threshold means fewer false alarms and more clipping. Lower means the reverse.
3. Set your silence duration: This is how long the silence must persist before the detector closes a speech segment. Silero's default min_silence_duration_ms is 100ms. OpenAI's documented server_vad example uses 500ms. It's the same concept with a 5x difference, and neither is wrong for the other's use case.
4. Set your padding: Silero's speech_pad_ms defaults to 30ms and pads each side of a detected chunk. OpenAI exposes prefix_padding_ms, shown as 300ms in its example config. Padding is your direct lever against front-end clipping.
5. Re-test at your real sample rate: Silero supports 8kHz and 16kHz. If your callers arrive over PSTN, validate at 8kHz. Numbers gathered on 16kHz studio audio do not predict narrowband phone behavior.
Pro tip: your defaults come from the wrapper, and the model's authors never chose them. A framework can ship Silero with silence and padding values many multiples of Silero's own, and the audio characteristics that work well for batch transcription are the opposite of what a live phone call needs.
Read the config your framework actually passes down before you blame the detector.
Voice Activity Detection Best Practices Worth Setting Up Early
- Measure at the audio layer: Speech-to-text timestamps collapse under network stress and drop trailing words. Frame-level analysis of dual-channel audio gives you millisecond truth.
- Record in stereo from day one: Caller isolated on one channel, agent on the other. Cekura records every call this way for exactly this reason. Overlap detection is trivial with two channels and guesswork with one.
- Make your timing parameters configurable per scenario: A healthcare intake line for elderly patients and a drive-thru order line need different rhythms. Hardcoding either into your engine costs you the other.
- Track percentiles: A p99 of 2.5 seconds means 1 in 100 callers had a conversation that felt broken.
- Test with the accents you actually serve: A detector trained on thousands of languages still meets your callers for the first time in production.
Three mistakes you should avoid:
- Trusting lab accuracy: Clean 16kHz WAV files predict nothing about a cellular call from a parking garage.
- Tuning threshold and silence duration together: Change one, measure, then change the other. Moving both leaves you unable to attribute the delta.
- Treating VAD config as set-and-forget: A codec change, a telephony migration, or a framework upgrade can move your defaults without a line of your own code changing.
Our Verdict on Voice Activity Detection
Voice activity detection is infrastructure you cannot opt out of, and it is the cheapest place in your stack to buy a better conversation. Getting the threshold, silence window, and padding right will do more for perceived quality than another prompt revision.
It also will not rescue a badly designed turn-taking layer. A better detector raises your ceiling, and the logic consuming its output sets your floor. If your agent talks over people, audit the silence window before you audit the model.
Alternatives make more sense in three places:
- For offline diarization or research segmentation, pyannote is a better fit than a real-time detector
- For microcontroller deployments, a commercial engine optimized for embedded targets may beat a general-purpose model
- For a production phone agent, a neural detector plus a semantic signal plus honest measurement is the combination that holds up
Cekura Makes VAD Testing Easier
Configuring a detector only takes about an afternoon, but proving it works across ten thousand messy calls is the actual job, and it is why we built Cekura.
Pre-production
- Infrastructure Suite: 18+ prebuilt scenarios covering hold behavior, audio quality, interruption handling, language support, packet loss, and rapid-fire phrases
- Accent testing against regional and non-native speech
- Multilingual testing across languages and code-switching
Infrastructure
- Barge-in, cough, and mid-turn silence injected into live call audio
- A rubric rule that fails the run when the agent interrupts the caller
- GitHub Actions so every prompt, model, or VAD config change reruns the suite before merge
Observability
- Voice activity detection is run on each channel of the stereo recording to flag every talk-over
- Response latency reported at p50, p95, and p99
- Speech-quality metrics are scored on production calls automatically
We plug into what you already run. Cekura connects natively to Retell, VAPI, ElevenLabs, LiveKit, Pipecat, Bland, and more, plus any custom stack over SIP or webhook.
Since call audio carries names, dates of birth, and account numbers, Cekura is also SOC 2-, HIPAA-, and GDPR-compliant for transcript redaction, role-based access, and audit trails.
Healthcare accounts for more than half our customer base, where a clipped digit in an appointment date is a rescheduled surgery. That is the standard we build the detector tests against.
Book a demo, and we will run your agent against the interruption suite on your own traffic.
Frequently Asked Questions
Is voice activity detection the same as speech recognition?
No, voice activity detection is not the same as speech recognition. VAD answers whether someone is speaking, returning a probability per audio frame, while speech recognition converts that audio into words.
Most stacks run VAD first and pass only the voiced segments to the recognizer, which cuts transcription cost and reduces errors from background noise.
What's the difference between VAD and endpointing?
The main difference between VAD and endpointing is scope. VAD classifies a single frame of audio as speech or non-speech, while endpointing decides whether the caller has finished their whole turn.
Endpointing consumes VAD output alongside transcript finality and semantic signals, so VAD is an input to the decision rather than the decision itself.
Does VAD work in noisy environments?
Yes, modern neural VAD works in noise, though accuracy degrades as the signal-to-noise ratio drops. Energy-threshold detectors fail here because loud background sound reads as speech, a limitation documented in the G.729 standardization work back in 1997.
Neural detectors like Silero handle noisy audio considerably better, and you should still validate against the noise profiles your callers actually phone from.
What VAD threshold should I use?
Start at 0.5 and tune from there against your own audio. Silero's maintainers describe 0.5 as a "lazy" default that works acceptably across most datasets while recommending per-dataset tuning.
Raise it for noisy environments to suppress false triggers, and lower it when missing the start of speech costs more than the occasional false alarm.
What is the best way to test voice activity detection?
The best way to test voice activity detection is to simulate messy callers against your live agent and measure at the audio layer rather than the transcript layer.
Cekura runs barge-in, cough, background noise, and packet-loss scenarios through your stack, then analyzes each channel of the stereo recording to report interruptions and latency percentiles. Transcript timestamps collapse under network stress, so frame-level audio analysis is the reliable signal.