ASR accuracy testing for multilingual voice agents means measuring how well speech recognition converts what a caller actually said into text, separately for each language, accent, and code-switched combination a deployment expects to handle. Cekura tests this across 30 or more languages, including dedicated support for 9 Indian languages, tracking transcription accuracy as one of its named evaluation metrics.
TL;DR
- ASR accuracy testing measures speech-to-text quality separately per language, accent, and code-switched combination, not as a single aggregate score.
- Word Error Rate counts substitutions, deletions, and insertions against a reference transcript; Character Error Rate is the better choice for tonal and morphologically complex languages where word boundaries aren't stable.
- A transcription error and an intent error are not the same failure: wording can shift while the detected intent stays correct, so downstream accuracy needs its own check.
- Cekura tests across 30 or more languages, including dedicated support for 9 Indian languages, with transcription accuracy as one of its named evaluation metrics.
- Clean benchmark datasets routinely miss the failures that show up in real calls; background noise, accents, and code-switching all degrade accuracy in ways a curated test set doesn't reveal.
What Metrics Actually Measure ASR Accuracy?
Word Error Rate is the standard metric for ASR accuracy. It counts the substitutions, deletions, and insertions needed to turn a system's transcript into the reference transcript, divided by the total words in that reference. The formula works well for languages with clear word boundaries, English among them. It breaks down for languages where word segmentation itself is ambiguous. Character Error Rate solves this by counting at the character level instead of the word level, which is why it holds up better for tonal and morphologically complex languages. Research comparing the two found CER correlates more closely with human judgment than WER across linguistically distinct languages, Malayalam, English, and Arabic among them, and recommended prioritizing it in multilingual evaluation rather than treating WER as the default everywhere. A developer's guide to Cekura's evaluation metrics covers this same word-level comparison, plus why a strong WER alone still isn't the whole picture of accuracy. A broader look at voice agent accuracy across every layer covers ASR alongside intent, entity extraction, and task completion; this post goes deeper on the ASR and multilingual slice specifically.
Why Isn't WER Enough for Multilingual Agents?
A transcription error and an intent error are not the same failure. Treating them as one hides what actually matters to a caller. A wording shift that leaves the detected intent correct is a cosmetic miss. A wording shift that changes what the agent thinks the caller asked for is a real one. This split matters more in multilingual deployments specifically, because error rates already vary by language before intent is even scored, so a single blended accuracy number obscures two separate problems: how well the agent hears, and how well it understands what it heard. Cekura scores transcription accuracy and intent as separate named metrics for exactly this reason, rather than folding both into one number. Testing both layers separately is what actually predicts whether a caller gets the right outcome, not WER alone.
How Do You Test Code-Switching?
Testing code-switching means building test cases where a caller mixes two languages inside one sentence, not just running each supported language separately in isolation. Code-switching is one of the hardest conditions for ASR to handle, and a monolingual-only test suite never exercises it. The failure is uneven across systems. Research benchmarking frontier ASR models on code-switched speech found the strongest models handle a language switch mid-sentence with only a small accuracy penalty relative to monolingual speech, while weaker models degrade substantially more. Performance varies enough by language pair that a single global code-switching score would hide real gaps. Cekura's own platform is built to simulate exactly this, mixing languages inside a single conversation rather than testing each language only in isolation. A multilingual test suite needs code-switched examples for every language pair a deployment actually expects, not just a token example or two.
How Does Cekura Test ASR Accuracy Across Languages?
Cekura tests voice agents across 30 or more languages, including dedicated support for 9 Indian languages, rather than treating one language as the reference and others as edge cases. Cekura's own multilingual testing coverage extends past the language list itself into configurable personas and background conditions, so a language isn't tested with one flat voice but across the range of speakers and environments an agent will actually meet. Accent handling gets its own layer. Cekura's accent and dialect testing lets a team compute accent-specific WER and CER so it can compare fairness across accents directly, instead of relying on one blended score that a strong majority accent could quietly prop up. Transcription accuracy is one of Cekura's own named evaluation metrics, scored as part of the same suite that covers intent, tool calls, and task completion, not a separate report on its own schedule. Cekura's own benchmarking covers the full voice stack under realistic conditions, accents, background noise, fast or hesitant speakers, not the speech recognizer alone in isolation.
Why Do Clean Benchmarks Miss Real-World Failures?
Public ASR benchmarks are built from curated, relatively clean audio. A system that performs well on one says less than it seems to about how that same system performs on an actual phone call. WildASR, a diagnostic benchmark built from authentic human speech, found that ASR systems near-human on curated benchmarks still fail under real-world conditions those benchmarks don't systematically cover, with degradation that varies by language and doesn't transfer from one condition to another. The same research found a sharper problem underneath the accuracy drop: under partial or degraded audio, a system can fabricate plausible-sounding words instead of failing visibly. That is a safety issue as much as an accuracy one. Public leaderboards that score models on standardized datasets are a reasonable starting point for picking an ASR engine, but they test the engine in isolation, not the conditions a specific deployment's callers will actually produce. Cekura's own benchmarking runs under those realistic conditions directly, for exactly this reason.
What Mistakes Do Teams Make When Testing ASR Accuracy for Multilingual Agents?
Testing ASR accuracy for a multilingual voice agent goes wrong in a few consistent ways.
- Reporting one blended WER instead of a per-language breakdown. A strong majority language can hide a much weaker minority language inside the same average.
- Skipping code-switching entirely. Callers who speak two languages naturally mix them mid-sentence, and a monolingual-only test suite never exercises the failure mode that actually happens in production.
- Testing only clean, studio-quality audio. Real calls carry background noise, phone compression, and accented speech that a curated benchmark filters out by design.
- Treating a transcription error and an intent error as the same problem. A wording change that leaves intent correct is a different failure than one that changes what the agent thinks was asked.
- Never revisiting the corpus. A test set built once at launch stops reflecting the languages, accents, and phrasing a deployment's actual callers use a year later.
Cekura's own per-language metric breakdown and code-switched scenario testing exist specifically to catch the first two mistakes before they reach production.
FAQ
What's the difference between Word Error Rate and Character Error Rate?
Word Error Rate counts substitutions, deletions, and insertions against a reference transcript at the word level, which works well for languages with clear word boundaries. Character Error Rate counts the same kinds of errors at the character level instead, which holds up better for tonal or morphologically complex languages where word segmentation itself is ambiguous. Multilingual evaluation research recommends reporting CER alongside WER rather than defaulting to WER everywhere.
Does a lower Word Error Rate always mean a better voice agent?
No. A transcript can have a nonzero WER and still preserve the caller's intent perfectly, and a transcript with a lower WER can still misroute a request if the specific words it got wrong happened to be the ones that mattered. Intent accuracy and entity accuracy need their own checks alongside WER, not as a substitute for it.
How do you test code-switching in a multilingual voice agent?
Build test cases where a caller mixes two languages inside one sentence, one set for every language pair the deployment actually expects, not a single generic example. Performance on code-switched speech varies significantly by model and by language pair, so a test suite needs coverage broad enough to catch a weak pair rather than assuming strong performance on one pair generalizes to the rest.
Does Cekura test ASR accuracy for multiple languages?
Yes. Cekura tests voice agents across 30 or more languages, including dedicated support for 9 Indian languages, and tracks transcription accuracy as one of its named evaluation metrics alongside intent, tool calls, and task completion. Accent-specific testing is handled as its own layer, exporting per-accent error rates rather than one blended score.
Why do ASR systems that score well on public benchmarks still fail in production?
Public benchmarks are built from curated, relatively clean audio, and strong performance there does not predict performance on an actual phone call with background noise, accented speech, or code-switching. Diagnostic research built from authentic real-world speech has found that ASR systems near-human on curated benchmarks still degrade under conditions those benchmarks don't systematically cover, and degradation from one condition or language doesn't reliably predict degradation in another.