Knowledge base accuracy testing for voice bots checks two things separately: whether retrieval found the right passage, and whether the spoken answer matches a known correct answer. It needs a question set with an answer key and calls made as audio. Cekura places simulated calls and scores each transcript against your knowledge base files and your answer key.
TL;DR
- Retrieval and answering fail separately, so score them separately. The Ragas and ARES frameworks both score context relevance and answer faithfulness as distinct dimensions.
- Retrieval does not guarantee a correct answer. On CRAG, a factual question answering benchmark of 4,409 questions, state-of-the-art industry RAG solutions answered only 63% of questions without any hallucination, and accuracy was lower on facts that change over time (arXiv 2406.04744).
- Voice adds an error before retrieval starts. In an EMNLP 2026 study of synthesized spoken multi-hop questions, corrupted entities in the transcribed query accounted for 87% to 96% of degraded cases on 2WikiMultiHopQA (arXiv 2608.22872).
- Real phone audio tends to raise word error rate. Per Cekura’s speech-to-text benchmark, 16 of 19 models scored a higher word error rate on eight real call recordings than on 1,000 public clips, but eight recordings is a small sample and many 95% ranges overlap.
- Cekura generates evaluators grounded in your uploaded knowledge base, scores each call with a Hallucination metric against those files, checks the answer key with Expected Outcome, and keeps scoring production calls.
What is knowledge base accuracy testing for voice bots?
Knowledge base accuracy testing for voice bots is a test method that asks a voice agent questions whose correct answers live in its documents, then scores what the agent said against an answer key. It covers the whole path a spoken question takes: speech-to-text, retrieval, the language model’s answer, and the text-to-speech voice that reads the answer out.
A hallucination check asks whether the answer is supported by the documents. An accuracy test also asks whether it is the right answer for this caller, today. A bot that quotes last year’s price list is grounded and wrong.
Cekura runs these tests as simulated phone calls. You upload the knowledge base to the agent as PDF, TXT, JSON, MD, CSV or XML files, and Cekura uses it to generate evaluators grounded in your content, per its agent setup docs. The guide to hallucination detection for voice AI covers why agents make answers up; an accuracy test adds the answer key, retrieval metrics, spoken numbers and regression runs.
How does a voice bot give wrong answers from its knowledge base?
A voice bot gives wrong knowledge base answers through six failures: retrieval misses, wrong chunks, stale documents, grounded-but-wrong answers, made-up fill-ins and misread numbers. Each needs its own check, because the first two are search problems, the third is a content problem and the last three happen after retrieval. Cekura’s Hallucination metric compares each reply with the uploaded knowledge base files, so it flags fill-ins and guesses but passes a reply that matches an outdated or neighbouring passage in those files, which is why an answer key sits beside it.
| Failure | What the caller hears | Check that catches it | What the check needs |
|---|---|---|---|
| Retrieval miss | A guess, or “I don’t have that” for a documented fact | Context recall | The passage that holds the answer |
| Wrong chunk | The answer to a neighbouring question, such as the family plan price instead of the single plan | Context precision, answer correctness | Retrieved chunk IDs, answer key |
| Stale document | Last quarter’s fee, read confidently | Answer correctness after every document change | A dated answer key, plus a review-by date and an owner for each document |
| Grounded but wrong | A faithful summary of a passage that does not apply to this caller | Answer correctness, not groundedness | An answer key per caller type |
| Made-up fill-in | A plausible detail no document contains | Groundedness against the knowledge base | The knowledge base files |
| Number misheard or misread | “Fifteen” heard as “fifty”, or a dropped digit | Transcription accuracy, an audio judge | The call recording |
The stale-document row deserves its own test. CRAG, a NeurIPS 2024 factual question answering benchmark of 4,409 questions with mock web and knowledge graph search, found lower accuracy for facts with higher dynamism, meaning facts that change over time, and for long-tail entities and complex questions (arXiv 2406.04744). Company prices, opening hours and policy dates change too, so tag them and rerun their questions after each update.
Which metrics separate retrieval accuracy from answer accuracy?
Retrieval metrics are scores that check whether the right passage reached the model, and answer metrics are scores that check what the model did with it. Keep the two apart, because one end-to-end score cannot tell you which half to fix.
| Metric | Layer | Question it answers | Inputs it needs | How Cekura scores it |
|---|---|---|---|---|
| Context recall | Retrieval | Did retrieval return the passage that holds the answer? | Reference answer or marked passage, retrieved context | A Python metric you write, reading retrieved document IDs sent as call metadata |
| Context precision | Retrieval | Were the relevant chunks ranked near the top? | Retrieved chunks in rank order, relevance labels | A Python metric you write, reading ranked document IDs sent as call metadata |
| Faithfulness or groundedness | Answer | Is every claim supported by the retrieved context? | Retrieved context, reply | Hallucination metric, which checks the reply against the uploaded files |
| Answer relevance | Answer | Did the reply address the question asked? | Question, reply | Relevancy metric |
| Answer correctness or completeness | Answer | Does the reply match the answer key? | Answer key, reply | Expected Outcome metric, with your answer key as the expected outcome prompt |
The Ragas authors, Shahul Es and colleagues, describe metrics that score retrieval, faithfulness and generation quality “without having to rely on ground truth human annotations” (arXiv 2309.15217). ARES, published at NAACL 2024, trains lightweight judges for context relevance, answer faithfulness and answer relevance, then corrects them with a few hundred human labels (arXiv 2311.09476). Microsoft’s RAG evaluator documentation frames groundedness as the precision side of an answer and response completeness as the recall side.
Reference-free scores are cheap to run, but a faithful answer drawn from a near-miss document can still pass them. Answer correctness against your own key catches it. The guide to voice AI evaluation metrics pairs the Hallucination and Relevancy metrics to tell retrieval failures from generation failures.
How do you build a test set of questions with known answers for a voice bot?
A knowledge base test set is a frozen list of caller questions, each paired with the passage that answers it and the answer a correct bot must give. Build it from what callers ask, not from what the documents cover.
- Pull real questions from call transcripts and tickets, with personal data removed.
- Mark the source passage for each question, so retrieval is scored on its own.
- Write the answer key as criteria, such as “states the 30-day window and the receipt requirement”.
- Add questions the documents cannot answer, and expect a refusal or a handoff.
- Tag questions whose answers change, such as prices and policy dates, to rerun after each document update.
Account-specific answers need matching records. Cekura’s test profiles give a simulated caller a fixed name and date of birth that match a mock record in your system, so a balance answer is checked against the right account.
How does RAG accuracy testing for voice AI agents work?
RAG accuracy testing for voice AI agents is end-to-end testing that sends each test question as audio, so speech-to-text errors reach the retriever as they would on a real call, a step text-only RAG evaluation skips.
An EMNLP 2026 study of four synthesized English accents found corrupted query entities accounted for 87% to 96% of degraded cases on 2WikiMultiHopQA (arXiv 2608.22872). Synthesized speech understated real errors: on 500 real Nigerian-accented utterances, WER averaged 28.9%, against 7.9% to 17.1% when synthesized.
Per Cekura’s speech-to-text benchmark, 16 of 19 models had a higher word error rate on eight real phone recordings than on 1,000 public clips. Eight recordings is a small sample with overlapping 95% ranges: read it as a direction, not a ranking.
Cekura’s Transcription Accuracy metric, once attached to an evaluator, compares the provider’s transcript of the simulated caller with the exact words Cekura synthesized, so a misheard product name is caught before retrieval is blamed. Simulated callers use synthesized voices, so score real production calls too.
How do you test spoken numbers, prices and dates?
Spoken-number testing is a check that the figure the caller hears matches the figure in the document. Numbers fail when digits are misheard, the model copies the wrong figure, or the voice reads the right one badly. Cekura’s voice agent workflow benchmark publishes one failure where the transcript captured a phone number correctly but a different number went to the tool.
Cekura checks numbers at three levels. Its Response Consistency metric flags an agent that repeats a caller’s detail back incorrectly. A Python metric compares digits in the transcript with the answer key at no credit cost. An LLM judge with voice recording turned on hears how “$1,250” was spoken, billed per audio minute.
Repeat each number question. In Cekura’s workflow benchmark (9 configurations, 82 scenarios, 3 retained repeats), PolyAI reached the expected outcome on 98.8% of calls but scored 87.8% on pass³, the stricter test, where all three runs of a scenario must pass the applicable scored rubric gates. Providers chose their own models, speech components and settings.
How do you regression test a voice bot when knowledge base documents change?
Knowledge base regression testing is a rerun of a fixed list of caller questions every time a document, prompt or model changes, with the answer key updated in the same change. Skip the key update and a correct new price fails; skip the rerun and a broken old answer reaches callers.
Cekura’s knowledge base connectors fetch content from a website with the Website Scraper, and each sync updates every file the connector created. The scraper reads static HTML only, so a help center that renders articles with JavaScript may sync without its article text. The post on knowledge base connectors and agentic retrieval covers the sync design.
Cekura reruns evaluators two ways. A cron job runs selected evaluators on a schedule and alerts on failures, which suits synced sources. A GitHub Actions workflow runs them on each push or pull request, which suits documents in a repository. Cekura’s docs suggest a 95%+ long-term pass rate, not 100%. The chatbot testing guide covers regression runs after knowledge base changes.
How do you monitor knowledge base accuracy on live calls?
Production knowledge base monitoring is the scoring of real calls after they end, with metrics such as Hallucination, so drift shows up in live traffic. Live calls have no answer key, so monitoring relies on groundedness and on patterns across many calls.
Cekura scores ingested production calls with the Hallucination metric against the agent’s knowledge base files, and Insights groups each metric’s failing calls into themes as they arrive. Cekura’s Deep Research covers what no metric was written for: it samples up to 500 calls by default from a chosen window, reads each transcript against the agent’s description, and returns failure modes ranked by urgency, each with a suggested fix and example calls. Invented confirmation numbers or IDs rank as critical, and a finding can become a new metric.
Deep Research has three limits. It is enabled per project. It reads transcripts, not audio, so a misread number still needs an audio judge. Each run costs 20 credits by default.
Frequently asked questions
What is the difference between knowledge base accuracy testing and hallucination detection?
Hallucination detection asks whether an answer is supported by the documents. Knowledge base accuracy testing also asks whether it is the correct answer for that caller. A bot that faithfully quotes an outdated policy passes the first and fails the second. Cekura covers both: the Hallucination metric checks support against the uploaded files, and Expected Outcome checks each answer against the criteria you wrote for that question.
How many test questions does a voice bot knowledge base need?
Start with 10 test questions, then grow the set from failures. Cekura’s docs suggest running them, fixing what fails and expanding, with 70% to 80% passing on a first run and 90% to 95% after refinement as realistic targets. For a knowledge base, cover the most-asked facts first, then every fact that changes over time, then several questions your documents cannot answer.
Can you run RAG accuracy testing for voice AI agents without retrieval logs?
Yes, but only at the answer layer. A black-box phone test scores groundedness against the knowledge base and correctness against an answer key, yet it cannot tell a retrieval miss from a model that ignored the right passage. If your pipeline logs retrieved document IDs, send them as call metadata through the Custom Integration webhook or the observability API. Cekura’s Python metrics read custom metadata fields, so recall against your marked passages is scored at no credit cost.
How much does knowledge base accuracy testing cost?
Cost depends on call minutes, metric choice and repeats. Cekura’s pricing page lists $0.25 per voice testing minute and $0.05 per monitored call on pay as you go, with 10+ standard metrics free. Cekura’s docs list the Hallucination metric at 0.6 credits per call, Expected Outcome at 0 credits and Python metrics at no credit cost. Price your own metric set from both sources, then multiply by the number of repeats per question.
Does Cekura test knowledge base accuracy for voice bots?
Yes. Cekura generates evaluators grounded in the knowledge base you upload or sync, runs them as simulated calls with a chosen caller voice, speaking pace and background noise, scores each transcript with the Hallucination metric against your files and with Expected Outcome against your answer key, and keeps scoring production calls after launch. Deep Research audits live calls for failure modes no metric covers. Book a demo to run it on your own documents.
