Across public clips and private recordings of real phone conversations, NVIDIA's new model made the fewest transcription errors of the 21 speech-to-text models we test, and it is the only one in the top five on both datasets.
NVIDIA Nemotron 3.5 Transcribe has the lowest overall word error rate on Cekura Bench: 2.28% across both of our datasets, or about one word in 44. Zoom Scribe Pro v2.1 is second at 2.35%, and Reson8 is third at 2.38%.
The result rests on consistency. Nemotron is fifth of 21 on our public dataset and second on our private one, a set of real two-person phone conversations that has never been published, so it can't have reached a model through public training data. No other model is in the top five on both.
That matters because the two datasets disagree about almost everything else. Zoom Scribe Pro v2.1, which leads on the public clips, is seventh on the private calls. Reson8, which leads on the private calls, is eighth on the public clips.
Why we keep a private test set
A benchmark measures how well a model generalizes only if the model hasn't seen the test. For speech that is hard to guarantee. Modern speech recognition models are trained on web-scale audio; OpenAI's Whisper, for one, was trained on 680,000 hours collected from the internet. A test set published on the open web can end up in training data, and a model that has trained on the test scores well for reasons that won't carry over to your callers. In machine learning this is called test-set contamination.
Cekura Bench uses two datasets for that reason.
The public set is Pipecat's speech-to-text benchmark: 1,000 short clips and 23,851 reference words, published on Hugging Face. Because it is open, anyone can reproduce our numbers on it, and anyone can train on it.
The private set is eight recordings from four real phone conversations, licensed from Ocular, with each speaker on a separate track. It holds 12,561 reference words. We don't publish it, so a score on it reflects how a model handles conversational audio it is hearing for the first time.
Conversation is also harder to transcribe than clean clips. Speakers overlap, restart sentences, trail off and drop in backchannels like "mm-hmm". The private set is where those effects show up in the numbers.
Nemotron makes the fewest errors overall
Across both datasets, 36,412 reference words in all, Nemotron 3.5 Transcribe got 831 wrong. Zoom Scribe Pro v2.1 got 855 wrong and Reson8 868. AssemblyAI Universal 3.6 Pro, third on the public clips, got 987 wrong.
On the private calls alone, Reson8 is first at 2.93% and Nemotron second at 3.03%: 368 errors against 381 across 12,561 words, well inside the 95% range of either model. GPT-4o Transcribe got more than 1,500 wrong.
Spread over an average conversation in the set, which runs to about 3,100 words, that is roughly 92 errors for Reson8, 95 for Nemotron and 141 for Universal 3.6 Pro. In a voice agent, every one of those errors goes straight into the language model's context. A date of birth with one wrong digit, or a medication name heard as an everyday word, becomes a failed lookup or a wrong answer, and nothing downstream flags it.
Word error rate on Cekura Bench's private dataset for 21 speech-to-text models, with 95% ranges. Lower is better.
The public-to-private gap
Putting each model's two scores side by side shows how far its public result carries over to unseen conversations.
Eighteen of the 21 models have a higher error rate on the private calls, and the median model's is 60% higher than on the public clips. The models at the top of the public leaderboard move furthest: Zoom Scribe Pro v2.1 goes from 1.55% to 3.87%, Zoom Scribe Pro from 1.75% to 4.83%, AssemblyAI Universal 3.6 Pro from 1.77% to 4.50% and Zoom Scribe from 1.83% to 5.17%, between 2.5 and 2.8 times their public error rates.
Nemotron's error rate rises by about the median amount, from 1.89% to 3.03%, but it starts low enough to stay near the top of both. Three models do better on the private calls than on the public clips (Smallest Pulse, Gradium and Deepgram Flux Multilingual), but all three sit above 3.4% on the private set.
Word error rate on the public Pipecat dataset and the private Ocular dataset for 21 speech-to-text models. Lower is better.
Side by side
The top three overall and the next three on the public set, with word error rate and rank out of 21:
| Model | Public (Pipecat) | Private (Ocular) | Overall |
|---|---|---|---|
| NVIDIA Nemotron 3.5 Transcribe | 1.89% (#5) | 3.03% (#2) | 2.28% (#1) |
| Zoom Scribe Pro v2.1 | 1.55% (#1) | 3.87% (#7) | 2.35% (#2) |
| Reson8 | 2.10% (#8) | 2.93% (#1) | 2.38% (#3) |
| Zoom Scribe Pro | 1.75% (#2) | 4.83% (#18) | 2.81% (#9) |
| AssemblyAI Universal 3.6 Pro | 1.77% (#3) | 4.50% (#16) | 2.71% (#6) |
| Zoom Scribe | 1.83% (#4) | 5.17% (#19) | 2.99% (#11) |
The three models just behind Zoom Scribe Pro v2.1 on the public clips rank 16th, 18th and 19th on the private calls. Ranked on either dataset alone, Nemotron would not be first; ranked on both, nothing makes fewer errors.
How we measure
- Same frozen audio. Every model gets identical files. The public set is pinned to a fixed revision, and each file is checked against its hash before it is sent.
- Word error rate. WER is substitutions plus deletions plus insertions, divided by the number of reference words. Before scoring, the reference and the transcript both go through the open-source Whisper English text normalizer, so formatting choices don't count as mistakes: "Dr." matches "doctor", "fourteenth" matches "14th", and fillers like "um" and "mm-hmm" are dropped on both sides.
- Pooled, not averaged. We sum errors across a whole dataset and divide by its total word count, so a long recording counts in proportion to its length. Overall WER pools both datasets the same way: errors on all 1,000 public clips and all eight private recordings, divided by their 36,412 reference words.
- Uncertainty. Each dataset score has a 95% range from 10,000 bootstrap resamples, drawn on the private-set chart as a thin line. On the private set we resample whole conversations rather than single tracks, because the two sides of one call are not independent samples.
- Accuracy comparison. Nemotron 3.5 Transcribe transcribed each recording as a complete file, so this comparison is about accuracy; our latency rankings cover the streaming models.
- Open harness, independent grader. The harness and the public data are open source on GitHub, and results are published under CC BY 4.0. Vendors see their results before publication and can report configuration errors, but they can't edit or veto them. Cekura doesn't build or sell speech-to-text models.
What this means if you're choosing a speech-to-text model
- Rank on audio like yours. Between our public and private sets the top three changed completely. Use a public leaderboard to build a shortlist, then decide on recordings that sound like your calls.
- Look for consistency as well as the top score. A model that ranks well on both clean clips and real conversations is a safer bet than one that leads on a single test set.
- Pair accuracy with latency for live agents. A live agent also needs the final transcript quickly once the caller stops talking. Our leaderboard ranks final-text latency for the streaming models separately.
- Test the whole agent on your own calls. Cekura runs your agent through simulated and real calls and measures what it heard, what it did and how fast it answered.
Full results for all 21 models, with head-to-head comparisons, are on the speech-to-text leaderboard. If you build a speech-to-text model, send us your endpoint and we'll run it on the same audio, with the same scoring, as every other model. Congratulations to the NVIDIA team on the release.
FAQ
What is the most accurate speech-to-text model on Cekura Bench?
NVIDIA Nemotron 3.5 Transcribe has the lowest overall word error rate of the 21 models tested, at 2.28% across the public and private datasets combined. Zoom Scribe Pro v2.1 is second at 2.35% and Reson8 is third at 2.38%.
What is the most accurate speech-to-text model on real phone calls?
On Cekura Bench's private dataset of real phone conversations, Reson8 has the lowest word error rate, at 2.93%. NVIDIA Nemotron 3.5 Transcribe is second at 3.03% and Cartesia Ink 2 is third at 3.09%, and their 95% ranges overlap.
Why does Cekura Bench use a private dataset?
Public test sets can end up in a model's training data, which inflates its score on them. Cekura Bench's private recordings have never been published, so a score on them reflects how well a model transcribes conversational audio it is hearing for the first time.
What is in Cekura Bench's private dataset?
Eight recordings from four real two-person phone conversations, licensed from Ocular, with each speaker on a separate track. Together they contain 12,561 reference words. The recordings are not published, and their 95% ranges are computed by resampling whole conversations.
How does Cekura Bench calculate word error rate?
Word error rate is substitutions plus deletions plus insertions, divided by the number of reference words. Both the reference and the transcript are normalized with the Whisper English text normalizer first, and errors are summed across the whole dataset before dividing. Overall word error rate sums errors and reference words across both datasets.
How did Nemotron 3.5 Transcribe do on each dataset?
It scored 1.89% on the 1,000 public Pipecat clips, fifth of 21, and 3.03% on the private Ocular calls, second of 21. It is the only model in the top five on both, and its error rate across the two datasets combined is the lowest of any model.