A voice agent can ace a benchmark and still fail the call.
In the transcript, it books the right appointment. On the phone, it talks over the caller, mishears a date of birth and goes quiet while a tool runs. The caller hangs up.
That gap is why we built Cekura Bench, and it's the simplest way to explain how it differs from τ-bench (tau-bench), the most cited benchmark for customer-service agents.
What τ-bench gets right, and where it stops
τ-bench, from Sierra, moved agent evals from single-turn Q&A to full conversations: a simulated user, a policy, real tools, and tasks across airline, retail, telecom and banking. Each task ends in a pass/fail check of the outcome (evaluation docs).
This year it added a voice mode that streams simulated speech into realtime models from OpenAI, Google, xAI and others (voice docs). A genuine step forward.
But it still tests the model, not the agent you ship. Your platform, prompt and tools never enter the picture, and you can't choose when the caller interrupts.
1. We simulate the whole call, in audio
Every test is a multi-turn simulated call over a real call path, so the score covers everything a caller hits: turn detection, speech-to-text, the model, tools, text-to-speech and transport.
Most production agents are cascades on Vapi, Retell, ElevenLabs, LiveKit, Pipecat or Telnyx. Cekura Bench runs against all of them, and against realtime models directly: our speech-to-speech board compares 12 realtime models, plus a cascaded baseline, across 3,198 live calls.
Callers are built to be difficult: background noise, accents, coughs, packet loss, silence, interruptions, changes of mind, prompt injection. There are 82 scenarios (59 appointment booking, 23 Medicare intake), each run three times. We report the pass rate and the all-three-pass rate.
2. Our caller is an agent, not a prompt
τ-bench's caller is an LLM improvising from a task description. A model decides when it interrupts, glitches land at random, and it can hallucinate: the voice mode reruns a call when it detects that (CLI reference).
Our caller uses an LLM too, inside a harness built on conditional actions, which we pioneered. A test is a set of rules: when the agent does X, the caller does Y.
-
Triggers. A rule fires on a moment ("the agent asks for your date of birth") or an exact phrase.
-
Fixed lines. When wording matters, the caller says it word for word.
-
Exact interruptions. "Cut in 1.2 seconds into the reply" happens at 1.2 seconds, every run.
-
Placed faults. Dead air, keypad tones, pre-recorded audio and network issues land where the test says.
So an agent that fails an interruption test failed at the same moment every other agent was tested on.
3. We measure the call, not just the outcome
Each call is scored on its own from the recording, transcript and tool calls, never from what the agent says about itself. Across our boards:
-
Task completion. A call passes only if every check passes.
-
Tool calls. Every expected call and argument, scored strictly and on the stored record.
-
Interruptions. Does it let the caller finish, yield when talked over, ignore a "mm-hmm"? Our turn-taking benchmark scores this across six phone line conditions.
-
Gibberish and transcription. Is the speech coherent, and did it hear the caller right?
-
Infrastructure. No-connects and dead air, kept apart from business logic.
-
Latency. Median and P90, measured at the caller's ear.
Every pass rate ships with a 95% confidence interval. Overlapping rows are a tie.
4. We don't sell the agents we grade
Sierra sells customer-service agents, so τ-bench's author also competes in the market it measures. That's no knock on the people who built it; it's an incentive.
Cekura doesn't build or sell voice agents or models. So the rules hold up even if you don't trust us:
-
Fixed configs. Locked before scoring, never changed after seeing results. Published dataset versions are frozen.
-
Public tests. Every scenario, script, expected outcome and tool call is on Hugging Face, versioned.
-
Nothing dropped. Failed and stalled calls are counted and flagged.
-
Recomputable. Audio, event logs and raw frames are kept, and the reference agent is one readable file.
Side by side
| τ-bench | Cekura Bench | |
|---|---|---|
| Built by | Sierra, which sells customer-service agents | Cekura, which sells no agents or models |
| Test | Text conversations, plus a voice mode | Simulated calls, audio end to end |
| Caller | LLM improvising from a task | LLM in a rule-based harness: fixed lines, interruptions and faults at exact moments |
| What it runs | Models, through their APIs | Realtime models and deployed agents on Vapi, Retell, ElevenLabs, LiveKit, Pipecat, Telnyx |
| Domains | Airline, retail, telecom, banking | Appointment booking, Medicare intake |
| Score | Pass/fail per task, in text and voice | Task completion, tool calls, interruptions, gibberish, transcription, infrastructure, latency |
| Reliability | pass^k (leaderboard shows Pass^1) | 3 runs per scenario, all-pass rate, 95% confidence intervals |
| Reproducibility | Open code, public leaderboard | Open code, versioned public dataset, raw audio and frames kept |
The question your customers actually ask
τ-bench answers whether a model can do the task. Cekura Bench answers whether your agent can do it on a phone call, with a real-sounding caller, three times in a row.
The speech-to-speech and voice agent workflow boards are live at benchmarks.cekura.ai. Run the same suite against your own agent from the open repo, and if you build a model or a voice platform, send us your config. We'll benchmark it exactly the way we benchmark everyone else.
FAQ
What is the difference between Cekura Bench and τ-bench?
τ-bench tests whether a model can complete a customer-service task, in text and in a newer voice mode. Cekura Bench tests whether a deployed voice agent completes the task on a simulated phone call, in audio end to end, three times in a row, with interruptions and faults placed at exact moments.
Does τ-bench test voice agents?
τ-bench has a voice mode that streams simulated speech into realtime models from OpenAI, Google and xAI. It tests the model through its API, so your platform, prompt and tools are not part of the score.
Which platforms does Cekura Bench test?
Cekura Bench runs against deployed agents on Vapi, Retell, ElevenLabs, LiveKit, Pipecat and Telnyx, and against realtime models directly. The speech-to-speech board compares 12 realtime models plus a cascaded baseline.
How does Cekura Bench measure reliability?
Cekura Bench runs each of its 82 scenarios three times and reports the pass rate and the all-three-pass rate (pass^3). Every pass rate ships with a 95% confidence interval, so overlapping rows count as a tie.
Is the Cekura Bench dataset public?
Yes. Every scenario, script, expected outcome and tool call is in a versioned dataset on Hugging Face, and the harness and reference agent are in the open cekura-bench repo.