How Voice Agents Know When a Caller Has Stopped Talking

A voice agent knows a caller has stopped talking by inferring it from two imperfect signals: voice activity detection, which hears sound but not words, and streaming speech-to-text, which delivers words late. On most turns the provider's final transcript settles it. When that final stalls, Cekura's voice agent checks the close against the call's own buffered audio.

TL;DR

• Turn-end is inferred from two signals, and neither is sufficient: VAD hears sound but not words, and streaming STT delivers words late, re-sends them, or occasionally delivers nothing.
• Interrupting a caller costs more than making one wait, so a fixed endpointing timer only chooses which mistake the agent makes.
• When the stream misbehaves, Cekura's voice agent re-transcribes the last stretch of buffered audio in one batch pass, and a gate checks the audio that arrived after that decode before any close is allowed.
• In ten days of Cekura production traffic, the turn-end arbiter ended about 6 turns in 100; the rest closed the ordinary way.
• Fixing the order of one rescue step cut committed guesses from roughly 1 in 50 rescued turns to roughly 1 in 3,000.

What does voice activity detection get wrong about turn-end?

Voice activity detection is a small model that emits speech-start and speech-stop events as audio arrives. Silero VAD, a widely used open model, processes one audio chunk of 30 ms or more in under 1 ms on a single CPU thread. VAD answers "is this sound speech-shaped", and it is wrong in both directions:

• Faint, speech-shaped background noise scores as a voice. Because a "still speaking" verdict vetoes ending the turn, the agent holds the turn open for a room that is merely not silent.
• Quiet trailing speech scores as silence, so the agent concludes the caller stopped while they were still going. On one Cekura call, a caller repeated an entire question and VAD produced no event at all.

VAD never knows what was said. It cannot tell a finished sentence from a caller who paused to find a word, which is the distinction a turn-end decision actually needs.

What does streaming speech-to-text get wrong about turn-end?

Streaming speech-to-text delivers provisional interim transcripts, revised as audio arrives, then a final transcript once the provider's own endpointing decides the utterance is over. STT knows what was said, which VAD never can. Cekura's voice agent sees three failures from streaming STT in production:

• Finals are sometimes held back for seconds after the caller stops.
• Stale interims get re-emitted and look like fresh speech.
• Occasionally a stretch of audio produces no transcript at all.

Provider-side end-of-speech events inherit the same blind spot. Deepgram's UtteranceEnd documentation shows the event firing after a 2-second gap between words "even though speech continues", because a gap between transcribed words is not proof of a finished thought.

Turn-detection models, such as the turn detector in LiveKit Agents, predict end of turn from the meaning and acoustics of speech on top of VAD. A better guess about a pause still does not help when the words themselves are late or missing, because an agent cannot answer text it does not have.

Why do voice agents interrupt callers mid-sentence?

Voice agents interrupt callers mid-sentence usually because the transcript runs behind the audio: words stop arriving, the agent concludes the caller has finished, and it replies while they are still talking. The two ways of getting turn-end wrong do not cost the same:

• Waiting too long gives the caller dead air. They repeat themselves or wonder whether the line dropped. That is recoverable.
• Cutting in too early answers half a question. The second half arrives as a new turn, and the exchange has to be repaired. Often the caller gives up and starts over. That is not latency; it is a lost turn.

Ekstedt and Skantze's Voice Activity Projection frames turn-taking as prediction: a self-supervised model learns to project both speakers' upcoming voice activity, and from that predicts turn-shifts and backchannels without labelled data. A production agent adds one more constraint: before it acts on a prediction, it has to confirm that the caller has actually stopped.

For a longer treatment of the timing itself, Cekura's guide to endpointing and turn detection covers the three signals that trigger a turn.

Why can't a fixed endpointing timer fix turn-end detection?

A fixed endpointing timer cannot fix turn-end detection because it picks one point on the trade between interrupting and waiting and lives with it. Shortening the pause interrupts more callers. Lengthening it makes every caller wait, including the large majority whose finals arrived on time. Neither setting makes the agent better informed.

Final-transcript timing also varies by model. Per Cekura's Converse-STT benchmark, P95 final-text delay after speech ends ranges from 65 ms (Inworld STT-1) to 1.97 s (GPT-4o Transcribe) across 15 models on the same 206 conversational turns, each measured under its own provider's finalization contract. A timer sized for the slow end of that range makes every caller on a fast model wait; a timer sized for the fast end cuts in on a slow one. Production adds its own tail: Cekura's voice agent has seen finals held back for seconds after the caller stops.

The problem mirrors handling LLM stalls in voice agents. There, the fix was to bound an open-ended wait. Here, bounding the wait is the easy half. The hard half is being right, and the only evidence for that is the audio of the call itself.

What turn-end failures do callers actually hit?

Turn-end failures reach callers as late answers, cut-off replies, long silences, or an "are you still there?" prompt over their own speech. Cekura's voice agent handles tens of thousands of calls a week, mostly as the simulated caller that Cekura uses to test customers' voice agents. That makes Cekura's own caller a voice agent with the same turn-end problem as the agents it tests. Every row below happened in Cekura production traffic.

On one call, five and a half seconds passed between the caller's last word and the turn closing, on words the agent was already holding. Across the fleet, the longest single turn left open ran past a hundred seconds.

For a long time nothing in Cekura's voice agent owned the turn-end decision. Turn-end was the emergent product of finals, interims, VAD and a set of timers, each patched in to cover one race. The bug tracker showed the cost first: dozens of fixes over three months, filed under STT, turn-taking, transcripts and VAD, all traced back to that gap.

How does a batch re-transcription recover a stalled transcript?

Batch re-transcription recovers a stalled transcript by decoding the caller's recent audio a second time, as one complete clip instead of a live stream. On a healthy turn the provider's final lands within a fraction of a second of the caller stopping, and none of this runs. Cekura's voice agent reserves the batch path for turns where the stream does not do its job:

• a final that never comes
• an interim re-emitted after the caller has gone quiet
• a final that lands over a minute after the audio it describes
• a transcript that drifts from what was actually said
• nothing at all for a stretch of audible speech

Cekura's voice agent keeps a rolling buffer of roughly the last twenty seconds of caller audio, marked at each VAD start so a turn's utterance can be sliced out with a little pre-roll. When the stream stalls, that slice goes to the same provider's batch endpoint.

A streaming recogniser commits word by word, guessing at each moment. A batch recogniser has the end of the sentence available while it decodes the beginning, so it recovers utterances the stream dropped or truncated, on audio already captured and paid for once.

How does Cekura verify the caller has actually stopped before replying?

Cekura verifies the caller has stopped with a single gate that every turn-end request passes through, whether it comes from the arbiter, the ordinary end-of-turn logic or a watchdog. The arbiter fires when the stream has produced no new word for about two and a half seconds, so the clip it decodes ends at that moment. The gate then measures the loudness of whatever audio arrived after that point, against both the speech inside the clip and the call's own noise floor.

• Quiet means the caller stopped and the turn can close.
• Loud enough to be speech means the transcript was behind, not slow, and the caller is still going. The gate re-checks once, and if the audio is still live it extends the turn instead of closing it.

A close also has to leave a minimum tail of quiet, and the whole verification runs against a three-second budget, so the check can never become the thing holding the turn open. Without the gate, batch re-transcription is only a faster guess, and a faster guess is a machine that interrupts people more efficiently.

What did roughly 800,000 turn-end verification decisions show?

Cekura's verification gate logged roughly 800,000 decisions over ten days of production traffic, because it looks again whenever the picture changes. The figures below are decisions, not callers or turns; one turn can be checked more than once.

• About a quarter let a close through, most on a tail of audio that was already quiet.
• Around one in six held the close back because the tail still carried energy.
• One in eight extended the turn because the stream had moved on since the close was requested.
• About two in every thousand found live speech past the point the decode accounted for, and extended the turn rather than ending it.

That last group is exactly the interruption the mechanism exists to prevent: a caller still talking after the transcript suggested they had finished. The gate does not speed up a healthy turn. It bounds the bad turn, which is the one the caller remembers, and it is the same principle behind Cekura's approach to endpointing generally.

How often does Cekura's turn-end arbiter end the turn?

Cekura's turn-end arbiter ends about 6 turns in every 100 in live production, measured over ten days of traffic: roughly 5 where the words had already arrived and only the close was slow, and 1 where no final ever came. The rest close the ordinary way. That is the ratio the design wanted: a fallback that is genuinely a fallback.

Before the arbiter could act, it ran in shadow mode, logging the decision it would have made against what the pipeline actually did. Across roughly 82,000 shadow-mode turns, it would have closed about 1 turn in 40 sooner than the pipeline did. That is a different measure from the live figure: turns where the arbiter would have been faster, not turns it ended.

Three in four of the turns the arbiter would have closed sooner already had the words; only the close was slow. The remaining quarter, where no final ever came, held the entire extreme tail: nearly four seconds late on average, and over thirty seconds at worst.

Why did the interim-transcript fallback commit guessed finals?

The interim-transcript fallback committed guessed finals because it asked the provider for a final and gave up in the same instant. The fallback for a stalled interim asks the provider to flush whatever final it is holding, then commits a synthesised final from the interim text. Both happened at once. The provider's complete final landed a tenth of a second later, behind a guess the language model was already answering, and was absorbed as a duplicate. The fallback's own flush produced the answer it then ignored.

Cekura's voice agent now flushes a quarter of a second before the fallback's deadline and closes on the real final when one arrives. That one reordering is the largest measured change in the whole project:

• Before, a guessed final was committed on roughly 1 in 50 of the turns where the rescue armed.
• After, roughly 1 in 3,000.
• The flush now comes back with the provider's real words about eight times out of ten. The guess is what happens when it does not.

What happens when two voice agent recovery timers fire at the same moment?

When two recovery timers fire at the same moment, scheduling order decides which one wins, and the caller can be answered twice. In Cekura's voice agent, the interim fallback and the arbiter's no-final check were both armed at turn start with the same two-and-a-half-second deadline. Which fired first came down to about a millisecond of scheduling, and when the wrong one won, the caller got two replies.

Two attempted fixes preserved the race and only hid the margin. The real fix was to state precedence: the fallback owns a turn it has started rescuing, and the arbiter stands down until that rescue resolves or gives up.

"A rescue that looks like the thing it is rescuing will be mistaken for it."

Dileep Chagam, Founding Engineer, Cekura

Any team building recovery into live turn-taking will meet this collision. Two mechanisms aimed at the same moment need an explicit owner, never a race.

How can a recovered transcript cancel the voice agent's own reply?

A recovered transcript can cancel the voice agent's own reply when the pipeline mistakes it for new speech. On one Cekura call, a batch decode recovered a caller's words correctly. The pipeline treated that recovered final as fresh speech, opened a new turn, and fired barge-in handling, cancelling the inference its own text had just queued. The caller waited ten seconds and repeated themselves.

The same trap sprang again on that call over a comma. Echo suppression compared the streaming and batch versions of the sentence after stripping only trailing punctuation, so "yes, that is right" and "Yes. That is right." read as two different utterances.

In each case the system could not tell its own rescue work apart from the conversation it was rescuing. Barge-in handling is built to react to a caller talking over the agent; it has no way to know that the "caller" in this case was the agent's own recovery path.

What happens when speech-to-text stalls or returns no transcript in a voice agent?

When speech-to-text returns no transcript, the failure is silent: no error, no transcript, and no turn for any recovery path to attach to. Every recovery path in Cekura's voice agent was scoped to a turn, and turns were started by transcripts. The safety net was unreachable in exactly the situation it was built for, while the audio sat in the buffer.

The fix is to arm from the other signal. When VAD has reported speech and the stream has said nothing about it, that alone now schedules a batch decode. The decode is timed to land just before the idle timeout would fire, so a false VAD blip delays an "are you still there?" prompt by half a second at most and never cancels one. Any real transcript, resumed speech or bot speech cancels the decode first.

Over three days, this path armed about 4,600 times and decoded fewer than one arm in ten; the rest cost nothing, because a transcript or fresh speech arrived first. It recovered real speech in roughly eight calls per thousand.

Why would a voice agent ask "are you still there?" while the caller is talking?

A voice agent asks "are you still there?" mid-speech when its idle timer is keyed to transcript events rather than voice activity. On one Cekura call, the idle timeout counted ten seconds of "inactivity" while VAD was firing start and stop events every second, through roughly nine seconds of audible speech, and prompted over the top of the caller. The agent could hear the caller the whole time. It was not consulting the signal that was working.

This failure is invisible in a transcript, because the transcript is the part that stalled. It shows up only in the audio: two voices at once, a prompt landing on a caller mid-answer. Cekura's work on silence and overtalk detection scores runs from the audio channels rather than the transcript for exactly that reason.

Two gaps remain open in Cekura's voice agent. An STT provider returning no transcript for audio it accepted and billed is a vendor question, not something to engineer around indefinitely. And there are still occasional utterances where VAD produces no event on audio where, run offline against the recording, the same model plainly fires.

What should teams building voice agent turn-taking take from this?

Teams building voice agent turn-taking should treat turn-end as a decision the system owns, not a side effect of whichever upstream service goes quiet first. Cekura's voice agent reached six rules:

• Knowing when something has finished is a decision.
• A deadline tells you time has passed, not that you are right. When the two failure directions cost differently, the mechanism has to verify, not just expire.
• Use the evidence you already paid for. The audio was buffered regardless; a second decode costs a fraction of a second of provider time and was the cheapest evidence available.
• Run it in shadow before you let it act. Shadow data shows how much sooner you could close, never how often you would close too soon.
• Two recovery paths aimed at the same moment will collide. State precedence; never let scheduling order decide it.
• Never key your alarm to the signal that fails. The timer goes quiet exactly when you need it.

Cekura runs scenario suites as simulated calls that include callers who pause mid-sentence, callers who interrupt, slow speakers and background noise, and gates prompt and endpointing changes in CI against those scores. Earlier resilience lessons from the same work are in The Silence Between Words.

Frequently asked questions

How does a voice agent know when I have stopped talking?

A voice agent infers turn-end from voice activity detection and streaming speech-to-text. On a normal turn, the provider's final transcript settles it. When that final is late, missing, or contradicted by what the caller audibly said, a turn-end arbiter in Cekura's voice agent takes over and checks its guess against the call's buffered audio before the agent replies.

Why do voice agents interrupt people?

Voice agents interrupt people usually because the transcript is behind the audio. The agent sees words stop arriving, concludes the caller has finished, and replies mid-sentence. When Cekura's voice agent has to close a turn without a prompt final, it decodes the buffered audio in one batch pass and measures the audio that arrived after that decode. If that tail still carries speech, the agent extends the turn.

What happens when the speech-to-text service fails outright?

When speech-to-text fails outright, the failure is silent: no error and no transcript. Cekura's voice agent detects it from VAD instead. Speech was detected and the stream said nothing about it, which is enough to trigger a batch decode of the buffered audio. Over three days, that path recovered real speech in roughly eight calls per thousand.

Why would an agent ask "are you still there?" while I am talking?

An agent asks "are you still there?" while you are talking because the idle timer behind that prompt is keyed to transcript events rather than voice activity. When the transcript stalls, the timer sees silence even while VAD is firing. Cekura's voice agent once counted ten seconds of inactivity through roughly nine seconds of audible speech for this reason.

Does a longer endpointing pause stop a voice agent from interrupting?

A longer endpointing pause does not stop interruptions; it moves them. Every caller waits longer, including the majority whose finals arrived on time, and callers whose finals run later than the new threshold still get cut off. Per Cekura's Converse-STT benchmark, P95 final-text delay already varies from 65 ms to 1.97 s across 15 speech models on the same 206 turns, each measured under its own provider's finalization contract, so no single pause fits every model.

Building voice agents that have to survive real callers? This is the class of failure Cekura tests for.