Evals, short for evaluations, are systematic tests that measure how well an AI system performs a task. An eval pairs a set of inputs with a way of scoring the outputs, then reports an aggregate. Where a unit test asserts one exact result, an eval measures quality across many cases and tells you whether the system got better or worse.
Last updated: August 2026 By Atul Jain
Teams adopt evals for one reason. Once a language model sits inside a product, nothing else tells you whether a change helped. The code still runs, the tests still pass, and the answers quietly get worse.
What are evals?
A single test answers a specific question about a system's behaviour, in a way that produces a number you can compare over time. "Does the classifier assign the right ticket category?" is an eval. "Does the summary stay faithful to the source?" is an eval. "Does the app work?" is not.
The word covers three things people often mix up:
- One test: a dataset, a task, and a grader.
- A suite: the collection you run together.
- The practice: building them, running them, and acting on what they say.
The distinction matters because one test scoring well says almost nothing. A suite covering your actual traffic, run on every change, is what produces trust.
Why do AI systems need evals instead of ordinary tests?
Four properties break conventional testing.
Outputs vary between identical runs. Setting temperature to zero narrows this and does not remove it. A test that passed once may fail next time with no change to anything you control.
Correctness is a matter of degree. "Did it return valid JSON?" is binary. "Was the answer helpful?" is not, and most of what users care about is the second kind.
The system changes without you. Providers update checkpoints. A prompt reworded for tone shifts behaviour across the entire input distribution, and nobody sees it if the only check is someone reading a few outputs.
Failure is silent. A wrong answer returns a 200 and logs nothing. There is no stack trace, only a metric that drifts.
An empirical study of 99 reports written by students who built and deployed LLM-powered applications as part of a university course, published on arXiv in July 2025, found the reported challenges were integration failures, unpredictable outputs, prompt sensitivity, hallucinations, and uncertainty about correctness. Evals exist to convert that last item, uncertainty about correctness, into a number.
What are the components of an eval?
Four parts. Missing any one is what makes a suite untrustworthy.
The dataset. Inputs the test runs against, ideally sampled from real traffic. Each case may carry a ground truth label, or may not, depending on the grader.
The task. What the system is asked to do with each input, held fixed so that changes in the score reflect changes in the system rather than changes in the question.
The grader. How an output becomes a score. This is where the design work lives, and the next section covers the options.
The threshold. What counts as a pass, and what constitutes a regression. A suite without thresholds produces a dashboard nobody acts on.
What types of graders are there?
Graders fall into three groups, and a working suite uses all three in different proportions.
Code-based graders
Deterministic rules written in code: exact match, regex, JSON schema validation, numeric tolerance, "does the output contain the required disclosure", latency budget. They cost nothing per run, return instantly, and never disagree with themselves.
OpenAI's documentation describes string_check as a grader performing exact string matching against reference values, with test data supplied as JSONL where each line carries an input and a ground truth label. Read on 14 August 2026, that documentation also carries a deprecation notice: the platform goes read-only on 31 October 2026 and shuts down on 30 November 2026, which is worth knowing before building on it.
Use code graders for everything they can express. The most common waste in eval design is spending a model call on something a schema validator settles for free.
Model-graded evals
A second model scores the output against a rubric. This is the only practical way to measure faithfulness, tone or helpfulness at volume.
Anthropic's guidance names six concrete grading approaches: exact match, cosine similarity for semantic consistency, ROUGE-L for summarisation, an LLM rating on a Likert scale, LLM binary classification, and an LLM ordinal scale. Its eval design guidance also states three principles, one of which cuts against instinct: "More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals."
That principle is worth sitting with. The temptation is to build a small, carefully hand-scored set. The advice is the opposite: automate the grading so you can afford breadth, and accept slightly noisier signal per case.
Model graders carry their own biases. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge evaluated 15 LLM judges across MTBench and DevBench, covering 22 tasks and roughly 40 solution-generating models, producing over 150,000 evaluation instances. The authors report that position bias is not due to random chance, varies significantly across judges and tasks, and is strongly affected by the quality gap between the solutions compared, meaning judges are least reliable when two candidates are close in quality. The trade-offs of the pattern are covered in LLM as a Judge: How It Works, Pros, Cons, and Best Practices.
Human review
A person reads outputs and labels them. Slow, expensive, and still the ground truth every automated grader is calibrated against. Use it to build the labelled set your model grader is then checked against, not as the routine gate.
Offline and online evals
Offline runs happen against a fixed dataset, before release. They answer whether a change is safe to ship, and they are what belongs in continuous integration.
Online runs happen against live traffic, after release. They answer whether the system is behaving now, catching drift, unusual inputs and provider-side changes that a frozen dataset cannot see by construction.
Neither substitutes for the other, and the loop between them is what makes a suite improve: online monitoring surfaces a failure, and that failure becomes an offline case so it can never recur unnoticed. Cekura covers the production half of that loop in What Is LLM Observability? A Real-World Guide.
How is quality scored?
| Metric | What it measures | Grader type |
|---|---|---|
| Exact or schema match | Output structure and literal correctness | Code |
| Semantic similarity | Meaning preserved despite different wording | Code, using embeddings |
| Groundedness | Claims trace to the supplied context | Model-graded |
| Instruction adherence | The system stayed inside its brief | Model-graded |
| Refusal accuracy | Correct refusals, without over-refusal | Mixed |
| Pass rate over repeats | The case passes consistently, not once | Code wrapper |
| Latency at p50 and p90 | Response time users actually feel | Code |
| Cost per completed task | Token spend for a real unit of work | Code |
Attach a threshold to each metric you track. Gate on movement against the previous baseline rather than an absolute score, because an absolute gate gets switched off the first week it blocks a release.
Benchmarks are not the same thing
Public benchmarks such as MMLU or HellaSwag compare models in general. They are useful for narrowing a shortlist when choosing a model, and they say nothing about whether your prompt, your retrieved context and your output schema work together on your traffic.
A model that ranks higher on a public benchmark can perform worse in your application, because your prompt was tuned around the previous model's habits. Treat a benchmark result as a hypothesis. Your own test set is the evidence.
Common mistakes to avoid
- No thresholds. Numbers with no defined pass condition become a dashboard nobody reads.
- Grading with a model from the same family as the system under test. The judge inherits a preference for that family's style, and the preference points away from human judgment.
- Unversioned graders. When the judge model or judge prompt changes without a version bump, a score movement is uninterpretable, because you cannot tell whether the system changed or the ruler did.
- Testing only edge cases. If most traffic is three intents, most of the suite should be those three intents.
- Never adding incidents. A production failure patched in application code and never turned into a test case will happen again.
- Running once. A single pass on a non-deterministic system is an anecdote. Repeats are what turn it into a measurement.
What this looks like on a voice agent
Text evals assume the output is a string. On a voice agent it is not. The output is a conversation carrying speech recognition, turn-taking, interruption handling and a telephony leg, and any of those can fail while the language model behaves correctly.
Cekura evaluates that whole path rather than the model alone. Cekura simulates full conversations against a live agent, scores each turn and each call against configurable evaluators, and feeds production failures back into the test set. Cekura documents the metric set in A Developer's Guide to Voice AI Evaluation Metrics.
The repeat discipline is visible in Cekura's own measurement. Per Cekura's benchmarks, six voice orchestration platforms were tested with 59 evaluators across four categories, each scenario run three times, and scored on pass^3, meaning a scenario counts as passed only when all three runs pass. Retell scored 96.6%, Vapi 94.9%, Pipecat 89.8%, LiveKit 84.7%, Synthflow 81.4% and ElevenLabs 76.3%.
Two caveats travel with those figures wherever they appear. Pass^3 is deliberately harsher than a single-run pass rate, so the two are not comparable. And the language model was held constant at gpt-4.1 at temperature 0 across all six. Speech recognition was pinned to Deepgram nova-3 on Vapi, Synthflow, LiveKit and Pipecat only, because Retell exposes only a coarse mode and ElevenLabs forces its own Scribe, so recognition is an uncontrolled variable on those two. Most of the spread therefore describes orchestration behaviour rather than model choice.
Frequently asked questions
What are evals?
Evals are systematic tests that measure how well an AI system performs a task. Each one pairs a dataset with a grader that turns outputs into scores, so quality can be compared across model versions, prompt changes and time. The term is short for evaluations.
What is the difference between an eval and a unit test?
A unit test asserts one exact result and fails on any deviation. An eval scores outputs across many inputs and reports an aggregate, because the same input can produce different outputs and most quality questions have degrees rather than a single right answer.
What is an eval framework?
An eval framework is tooling that handles the mechanics: storing datasets, running cases against a model, applying graders, and tracking scores across runs. The framework is not the hard part. Choosing the right graders and writing criteria that reflect what users care about is the hard part.
Do you need an LLM to grade evals?
No, and often you should not. Code-based graders handle structure, required content, numeric tolerance and latency for free and without disagreeing with themselves. Reserve a model grader for qualities code cannot express, such as faithfulness or tone.
How many eval cases do you need?
Anthropic's guidance is to prioritise volume, on the basis that more questions with automated grading beat fewer questions graded by hand. The practical constraint is frequency: a set small enough to run on every change is worth more than a large one that runs quarterly, so automate the grading first and let that decide the size.
Do the Cekura benchmark numbers apply to text-only evals?
Not directly. The pass^3 rates from 96.6% for Retell down to 76.3% for ElevenLabs measure voice orchestration platforms with the language model held constant at gpt-4.1 at temperature 0 across all six, though speech recognition was pinned on only four of them, so they describe orchestration behaviour rather than model quality, and pass^3 is harsher than a single-run pass rate. The transferable idea is requiring repeated passes before a case counts as passing.
Cekura runs evals on voice and chat agents across whole conversations, from simulation through production monitoring and back into the test set. Book a demo to see it run against your own agent.
