LLM testing is the practice of checking that a language model behaves acceptably across many inputs, rather than checking that code returns one fixed value. A conventional test asserts a single right answer. An LLM test scores outputs against criteria, samples enough inputs to see the distribution, and fails when quality drops rather than when a string changes.
Last updated: August 2026 By Tarush Agarwal
That gap is where production incidents live. You can hold full unit test coverage on the code surrounding a model and still ship an agent that invents a refund policy, ignores a system instruction, or costs three times more after a checkpoint update you did not make and were not told about.
This guide covers the two jobs LLM testing is often confused between, the five layers a suite should contain, how deterministic and model-graded checks divide the work, how to build a test set worth trusting, and how to run all of it on every deploy.
What is LLM testing?
LLM testing evaluates a model's outputs against acceptance criteria across a representative set of inputs. It answers a different question from conventional QA. Conventional QA asks whether the system did what the code says. LLM testing asks whether the system did something a competent reviewer would accept, on inputs nobody wrote a branch for.
An empirical study of 99 reports written by students who built and deployed LLM-powered applications as part of a university course, published on arXiv in July 2025, found testing strategies combined manual and automated methods to evaluate both system logic and model behaviour, with exploratory testing, unit testing and prompt iteration among the most common practices. The challenges the authors recorded were integration failures, unpredictable outputs, prompt sensitivity, hallucinations, and uncertainty about correctness.
That final item is the honest state of the field. Teams frequently cannot tell whether an output is right, and a testing practice has to be designed around that rather than pretending it away.
Model-centric and application-centric testing are different jobs
Most confusion about how to test an LLM comes from collapsing two separate activities.
Model-centric evaluation asks which model is better in general. It runs public benchmarks such as MMLU, HellaSwag, TruthfulQA or SWE-bench, or reads a leaderboard ranking models by human preference. This is useful for procurement: it narrows the shortlist when you are choosing between models.
Application-centric testing asks whether your system, with your prompt, your retrieved context and your tools, behaves correctly on your traffic. Nothing on a public leaderboard tells you this.
The distinction has a practical consequence. A model that scores higher on a general benchmark can perform worse in your application, because your prompt, your context format and your output schema were tuned around the previous model's habits. Benchmark position is a hypothesis about your system. Your own test set is the evidence.
If you are testing LLM applications rather than shopping for a model, spend your effort on the second job. Public benchmarks are also increasingly contaminated by training data, so a rising score does not reliably mean a rising capability.
Why is testing an LLM harder than testing normal software?
Four mechanics differ.
Outputs are non-deterministic. The same prompt can return different text on consecutive calls. Setting temperature to zero narrows this but does not eliminate it. A single passing run proves very little.
Correctness is graded, not binary. Most answers are partly right. "Did the model refuse the out-of-scope request?" has a yes or no answer. "Was the summary faithful to the source?" does not.
The system under test changes without you. Providers update checkpoints. A prompt reworded for tone shifts behaviour across your whole input distribution, and the shift is invisible if the only check is a person reading a handful of outputs.
Failures are silent. A wrong answer returns HTTP 200. Nothing raises, nothing is logged as an error, and the only symptom is a user who does not come back.
What should an LLM test suite cover?
Five layers, in the order they usually repay the effort.
Correctness
Does the output carry the right facts and follow the instructions given? This covers format compliance, whether required fields appear, whether the model stayed inside the scope its system prompt defined, and whether claims are grounded in retrieved context rather than invented.
Regression
Does behaviour hold after a change? This layer has the clearest return, because it catches the failure nobody plans for: a prompt edit or a model version bump that improves one thing and quietly breaks another. Regression testing needs a frozen input set and a stored baseline, or there is nothing to compare against.
Safety and adversarial behaviour
Does the model refuse what it should refuse, under pressure? Single-turn probes understate this badly, because attacks that fail in one message often succeed across several turns. Budget for adversarial inputs written by someone trying to break the system, not by the person who wrote the prompt.
Performance and cost
Latency at p50 and p90, tokens consumed, and cost per completed task. These are the cheapest signals to collect and the ones most often missing, which is how a threefold cost increase gets discovered in a billing statement rather than a test report.
Integration
Does the surrounding system survive what the model returns? Malformed JSON, a tool call with a wrong argument type, a response exceeding a downstream field limit. The study above lists integration failures first among reported challenges, which matches how these systems break in practice.
How to test an LLM: deterministic checks and model-graded checks
Every workable suite mixes two kinds of check, and the split is most of the design work.
Deterministic checks
Code compares the output against a rule: exact match, regex, JSON schema validation, "does the answer contain the required disclosure", numeric tolerance, latency budget. These cost nothing per run, return instantly, and never disagree with themselves.
Use them for everything they can express. Teams routinely reach for an expensive model-graded check to test something a schema validator would settle for free.
OpenAI's documentation describes string_check as a grader performing exact string matching between model outputs and reference values, with test data supplied as JSONL where each line carries both an input and a ground truth label. One caution that most current writing on this topic has not caught up with: read on 14 August 2026, that same documentation carries a deprecation notice. The OpenAI Evals platform becomes read-only on 31 October 2026 and shuts down on 30 November 2026. Guides published before that notice still list it among recommended tools, so check its current status before you build on it.
Open-source frameworks covering the same ground include DeepEval, promptfoo, Ragas for retrieval pipelines and MLflow's evaluation module. They differ mostly in whether they optimise for a unit-test feel or a dashboard.
Why reference-based metrics like BLEU and ROUGE mislead
Older text metrics score an output by its overlap with a reference answer. BLEU, ROUGE and their relatives were built for translation and summarisation, where a reference exists and word choice is constrained.
They fail on open-ended generation for a simple reason: a correct answer phrased differently scores badly, and an incorrect answer reusing the prompt's vocabulary scores well. BERTScore improves on this by comparing embeddings rather than tokens, so paraphrase is handled, but it still needs a reference and still cannot tell you whether a claim is true.
Use them where a reference genuinely exists. For everything else, the choice is a deterministic rule or a model-graded check.
Model-graded checks and where they break
A second model scores the output against a rubric. This is the only practical way to measure faithfulness, tone, helpfulness or instruction adherence at volume, and it has documented failure modes.
Position bias is the best-studied one. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge evaluated 15 LLM judges across MTBench and DevBench, covering 22 tasks and roughly 40 solution-generating models, producing over 150,000 evaluation instances. The authors report that position bias is not due to random chance, varies significantly across judges and tasks, is only weakly influenced by the length of prompt components, and is strongly affected by the quality gap between the solutions being compared.
Read that last finding carefully. Judges are least reliable exactly when two candidates are close in quality, which is the comparison you most often need. The practical responses are to pin both the judge model and the judge prompt to versions, to run pairwise comparisons in both orders, and to write rubrics with concrete criteria rather than asking for a general quality score. For how the technique works end to end, see LLM as a Judge: How It Works, Pros, Cons, and Best Practices.
How to build the test set
A test set nobody trusts gets ignored, which is worse than not having one.
- Start from production traffic. Sample real inputs across the range you actually receive, including the boring ones. Hand-written cases cluster around what their author already thought of.
- Add every incident. A production fix is not finished until the triggering input is a test case. This is the highest-yield habit in the practice and the most commonly skipped, because incidents get patched in application code and never re-enter the suite.
- Test by capability, not only by sample. Held-out accuracy overestimates how well a model generalises, which is the argument behind CheckList, the behavioural-testing methodology from ACL 2020. It pairs linguistic capabilities against test types so gaps get found deliberately rather than stumbled into. In its user study, practitioners using it "created twice as many tests, and found almost three times as many bugs as users without it."
- Generate synthetic cases to fill gaps, then check them. A model can produce variations, paraphrases and adversarial rewrites of real inputs cheaply. Synthetic data covers the space; it does not tell you the expected answer, so a human still labels the ground truth.
- Cover the distribution, not only the edges. If 80% of traffic is three intents, most of the suite should be those three intents.
- Version it with the prompt. A score means nothing unless you know which prompt, which model and which test set produced it.
- Keep a holdout. Cases you iterate against stop measuring generalisation after a few tuning rounds.
Size matters less than expected. A well-chosen 150-case set that runs on every commit beats a 5,000-case set that runs quarterly.
How do you test a RAG application?
Retrieval changes the failure surface, so testing it as one block hides where the fault is. Split the two stages.
Retrieval is testable with conventional information-retrieval measures, because ground truth exists: for a given query, either the right document was returned or it was not. Track whether the correct chunk appears at all, and how highly it ranks.
Generation is testable for groundedness, meaning whether each claim in the answer traces to the retrieved context rather than to the model's own parameters. This is the check that catches a confident answer built from a document that was never retrieved.
Testing them separately matters because the fixes are unrelated. Poor retrieval is fixed with chunking, embeddings or reranking. Poor groundedness is fixed with the prompt. A single end-to-end score tells you something is wrong and nothing about which.
Testing LLM applications in CI/CD
Treat a prompt change exactly like a code change: it runs the suite, produces a diff against a baseline, and a human reads the diff before merge.
- On every pull request, run the deterministic layer and a small model-graded subset. It has to finish in minutes or people will route around it.
- On merge to main, run the full suite with repeats, and store scores against the commit.
- Gate on deltas, not absolutes. "No metric drops more than two points against the previous baseline" is enforceable. "Faithfulness above 0.9" gets switched off the first week it blocks a release.
- Re-run when nothing changed. Scheduled runs against a frozen input set catch provider-side drift, which is otherwise invisible.
- Budget the token cost. Repeats multiply it and model-graded checks add a second inference per case, so a suite of 200 cases at three repeats with one judge call each is 1,200 inferences per run. Price that before you wire it to every commit.
Multi-turn behaviour needs separate treatment, because a suite of single-message cases can pass while the conversation itself fails. Why Single-Turn Testing Falls Short In Evaluating Conversational AI covers where that gap opens.
What LLM testing looks like on a live voice stack
Text evals stop short of the hardest case. When a model sits inside a voice agent, the thing under test is not a string. It is a conversation carrying speech recognition, turn-taking, interruption handling and a telephony leg, and any of those can fail while the language model behaves perfectly.
Cekura tests that whole path rather than the model alone. Cekura simulates full conversations against a live agent, scores each turn against configurable evaluators, and replays real production calls against a new model version so a regression surfaces before the upgrade ships. Cekura documents that replay workflow in Test New Model Versions with Real Production Calls Using Cekura. Cekura also holds the monitoring half of the loop, described in What Is LLM Observability? A Real-World Guide, so production failures feed back into the test set instead of being patched and forgotten.
The method transfers, and it shows why repeats matter. Per Cekura's benchmarks, six voice orchestration platforms were tested with 59 evaluators across four categories, each scenario run three times, and scored on pass^3, meaning a scenario counts as passed only when all three runs pass. On that measure Retell scored 96.6%, Vapi 94.9%, Pipecat 89.8%, LiveKit 84.7%, Synthflow 81.4% and ElevenLabs 76.3%.
Two caveats travel with those figures wherever they are quoted. Pass^3 is deliberately harsher than a single-run pass rate, so the two are not comparable. And the language model was held constant at gpt-4.1 at temperature 0 across all six. Speech recognition was pinned to Deepgram nova-3 on Vapi, Synthflow, LiveKit and Pipecat only, because Retell exposes only a coarse mode and ElevenLabs forces its own Scribe, so recognition is an uncontrolled variable on those two. Most of the spread therefore describes orchestration behaviour rather than model choice.
The transferable idea is the pass^3 design itself. Requiring three consecutive passes is what separates a case that works from a case that happened to work once, and it applies to a text-only LLM test suite exactly as it does to a voice one.
Which metrics are worth tracking
| Metric | What it tells you | Check type |
|---|---|---|
| Schema or exact match | Output structure is valid | Deterministic |
| Required-content presence | A mandated disclosure or field appeared | Deterministic |
| Retrieval hit rate and rank | The right document was found, and found early | Deterministic |
| Groundedness | Claims trace to the supplied context | Model-graded |
| Instruction adherence | The model stayed inside its brief | Model-graded |
| Refusal accuracy | Correct refusals, without over-refusal | Mixed |
| Pass^k over repeats | The case passes consistently, not once | Deterministic wrapper |
| Latency p50 and p90 | Response time users actually feel | Deterministic |
| Cost per completed task | Token spend for a real unit of work | Deterministic |
Track a small number and attach a threshold to each. Twenty metrics nobody has a threshold for is documentation, not testing.
Frequently asked questions
What is LLM testing?
LLM testing is the process of evaluating a language model's outputs against acceptance criteria across a representative set of inputs, rather than asserting one expected value. It combines deterministic checks that code can settle with model-graded checks that score qualities such as faithfulness and instruction adherence.
How is an LLM test different from a unit test?
A unit test asserts one exact result and fails on any deviation. An LLM test scores outputs across many inputs and fails when an aggregate drops below a threshold. Because the same prompt can return different text on consecutive calls, an LLM test also needs repeats before a single result carries meaning.
How do you test an LLM without reference answers?
Use a rubric-based model-graded check, and pin both the judge model and the judge prompt so scores stay comparable over time. Run pairwise comparisons in both orders, because a systematic study of 15 LLM judges across two benchmarks found position bias is strongest when two candidates are close in quality, which is the case you usually care about.
Are public benchmarks like MMLU enough to test an LLM?
No. Benchmarks and leaderboards compare models in general and are useful for narrowing a shortlist. They say nothing about whether your prompt, your retrieved context and your output schema work together on your traffic. Treat a benchmark result as a hypothesis and your own test set as the evidence.
How many test cases do you need?
Fewer than most teams assume. A 150-case set drawn from real production traffic, covering common intents rather than only edge cases, and running on every commit, is worth more than a large set that runs rarely. Add every production incident as a new case.
Do the Cekura benchmark numbers apply to text-only LLM applications?
Not directly. The pass^3 rates from 96.6% for Retell down to 76.3% for ElevenLabs measure voice orchestration platforms with the language model held constant at gpt-4.1 at temperature 0 across all six, though speech recognition was pinned on only four of them, so they describe orchestration behaviour rather than model quality, and pass^3 is a harsher measure than a single-run pass rate. The transferable part is the method: requiring three consecutive passes before a case counts as passing.
Cekura runs this loop on voice and chat agents end to end, from simulated conversations through production monitoring and back into the test set. Book a demo to see it run against your own agent.
