I've run chatbot evaluations across hundreds of LLM-powered agents, from appointment booking flows to financial support workflows. The teams that ship reliable agents treat evaluation like any other part of the process. Here are 3 methods and 8 metrics you can use.
What Is Chatbot Evaluation?
Chatbot evaluation is the process of assessing how a conversational AI agent performs in real-world interaction scenarios. That covers instruction-following, context retention across turns, task completion, and how the agent handles off-script or adversarial inputs.
Testing and evaluation target different failure modes. Testing validates that specific flows execute correctly. Evaluation goes broader and scores response quality over many conversations against defined criteria over time.
3 Chatbot Evaluation Methods
Which methods you run and when depend on where you are in the development cycle.
Method 1: Human Evaluation
What it is: Human reviewers score chatbot responses against a defined rubric, reading conversation transcripts and rating individual turns or full conversations. They score things like response accuracy, tone, task completion, and policy adherence.
How it works: You define 3 to 5 scoring criteria, assign scores per turn or per conversation, and aggregate results across a sample. What you measure depends on the use case. A healthcare scheduling agent gets scored on appointment confirmation accuracy and HIPAA-safe phrasing. A financial support bot gets scored on policy accuracy and escalation judgment. The standard approach is pairwise comparison, where reviewers see two responses to the same input and pick the better one rather than assigning absolute scores. This reduces disagreement between reviewers and is the method behind benchmarks like Chatbot Arena and MT-Bench.
When to use it: Start with human evaluation when you need a ground truth dataset to calibrate an automated evaluator against. Without it, you're tuning a judge with no reference point.
It also works when the failure type requires domain knowledge that an LLM judge will miss, like regulatory language, clinical accuracy, or compliance edge cases.
Real example: Regulatory language in financial chatbots fails in ways that don't show up in accuracy scores. The CFPB has documented consumer complaints about chatbot responses that were factually correct but legally problematic under UDAAP standards. UDAAP covers federal rules against unfair or deceptive financial practices, and a response doesn't need to be wrong to run afoul of them. A human reviewer with compliance context can catch that distinction where automated scoring tends to fall short.
The tradeoff: Human evaluation doesn't scale to production cadence. You can use it to build a ground truth dataset or audit a specific release, but running it after every prompt change or model swap isn't realistic. That's the challenge automated methods are built to address.
Method 2: LLM-as-a-Judge
What it is: A second language model scores your chatbot's responses on a structured rubric, which replaces human reviewers for high-volume evaluation.
How it works: The judge receives each conversation turn plus the scoring criteria, and optionally a reference answer, then returns a score with a step-by-step explanation of how it got there.
There are three judging modes. Single-output scoring sends one response plus a rubric and returns a score between 0 and 1. Reference-based scoring adds a gold-standard expected output, improving consistency on tasks where there's a correct answer.
Pairwise comparison shows the judge two responses to the same input and asks which is better. This reduces noise by making the judge compare rather than calibrate against a fixed scale.
Zheng et al. (2023) found that GPT-4-as-a-judge reached the same level of agreement as human judges over 80% of the time. That's what put the method on engineering teams' evaluation stacks.
That accuracy depends heavily on prompt design. Criteria need to be explicit, scoring anchors need examples at each level, and chain-of-thought reasoning is worth requiring so you can see why a score came out the way it did.
When to use it: Use LLM-as-a-judge whenever you need evaluation at a volume or speed that makes human review impractical. Regression runs after prompt changes, A/B tests across model versions, and daily production sampling all fit.
It also works well for dimensions that map cleanly to a rubric, like instruction following, factual accuracy against a known knowledge base, or policy adherence.
Real example: Judge bias operates independently of prompt quality. The MT-Bench paper found that GPT-4-as-a-judge shows measurable verbosity bias across all tested models, which means it systematically scores longer responses higher regardless of accuracy.
Your scoring criteria need to account for that explicitly.
The tradeoff: LLM-as-a-judge handles high-volume evaluation efficiently at the turn level. As conversations get longer, context overload degrades scoring consistency, and the judge can miss failure modes that only emerge over a full multi-turn sequence.
That's where simulation-based evaluation fills the gap.
Method 3: Simulation-Based Evaluation
What it is: A user simulator plays the role of a production user, running scripted scenarios through your chatbot end-to-end over multiple turns. The unit of evaluation is the full conversation. Did the agent complete the task, hold context, stay in role, and manage off-script inputs?
How it works: You define a set of goldens. Each one is a standardized test case with a scenario description, a user persona, and an expected outcome.
The simulator generates turn-by-turn exchanges against your live agent, and a scoring layer measures that exchange against your metrics. Scenarios cover happy paths, edge cases, topic switches, and adversarial inputs.
The simulation loop runs in parallel across hundreds of test cases simultaneously.
After a prompt change or model swap, you run the full set on the new version and compare scores. Any quality regression, meaning a score drop after a change, shows up on particular test types, visible individually rather than buried in an overall score.
Two evaluation modes cover different failure types. Full-conversation evaluation measures the entire exchange against session-level criteria, including task completion, conversation completeness, and role adherence across all turns.
Sliding-window evaluation rates each turn using a window of prior turns as context. It's better suited for catching turn-level failures like a dropped reference or an off-topic response mid-conversation.
When to use it: Simulation-based evaluation fits any chatbot where things go wrong through the conversation rather than in a single response.
Appointment booking flows, support escalation paths, and multi-step transactional workflows are good candidates. Regression testing at volume rarely has a manual equivalent that fits a production release cadence.
Real example: Multi-turn conversations produce performance degradation that single-turn evals leave unmeasured. Research across 200,000+ simulated conversations found that top LLMs show an average 39% drop in generation task performance when moving from isolated prompts to multi-turn settings.
Most of that degradation only shows up when you evaluate the full conversation sequence end to end.
The tradeoff: Simulation quality depends on how well your test cases cover the failure space. A set that only covers happy paths gives you false confidence.
The test bench benefits from adversarial personas, topic-switching users, and edge cases drawn from production failure logs to surface failures earlier in the process.
Which Method Should You Choose?
Three variables determine which method fits. Your position in the development cycle, how often your agent changes, and what kind of failure you're trying to catch.
Choose human evaluation if:
- You're building a rubric from scratch and need ground truth scores to anchor automated evaluators.
- The failure mode requires domain knowledge (regulatory language, clinical accuracy, financial compliance) that an automated judge won't pick up reliably.
- A one-time pre-release audit of a production agent fits here, too.
Choose LLM-as-a-judge if:
- You need to evaluate at volume: daily production sampling, regression runs after every prompt change, or A/B tests across model versions.
- The criteria map cleanly to a rubric. Instruction following, factual accuracy, and policy adherence all work.
- Per-score audit trails with reasoning traces matter more than aggregate pass/fail counts.
Choose simulation-based evaluation if:
- Your agent handles multi-step workflows where failure builds through the conversation.
- Regression coverage after every prompt or model change is a priority, and manual prompting won't scale to cover it.
- Edge cases, adversarial personas, and topic switches are worth testing before they reach production users.
Teams that run all three tend to split them by cadence. Human evaluation fits a monthly or per-release rhythm. LLM-as-a-judge goes on every pull request. Simulation runs nightly.
Pick based on what you're trying to catch. Compliance and judgment issues point to human evaluation, turn-level regressions to LLM-as-a-judge, and anything that only shows up across the full conversation to simulation.
Chatbot Evaluation Metrics
Metrics are split into two categories that answer different questions. Conversational metrics score how the conversation performed, while operational metrics connect that performance to business outcomes.
Conversational Metrics
Turn Relevancy
Measures whether each response is relevant to the user's latest input, given the prior conversation context. It uses a score of 0 to 1 per turn, aggregated across the full conversation.
Low turn relevancy is where context drift (when the agent starts losing track of what was said earlier) first shows up before it affects task completion and user drop-off.
The score uses a sliding window over the n most recent turns. If your window is too narrow, you miss drift, too wide, and you score noise from turns that stopped being relevant.
Knowledge Retention
Measures whether the agent correctly uses information the user provided earlier in the conversation. The score is the proportion of assistant turns that apply prior user-supplied facts without contradicting or omitting them.
A user who states their account number on turn 2 shouldn't have to provide it again on turn 7. When they do, the agent's lost track of the conversation in a way that affects accuracy and user trust.
Role Adherence
Measures whether the agent stays within its defined scope across the full conversation. The score is the number of in-role turns divided by total assistant turns.
An agent instructed to handle appointment booking that starts giving medical advice mid-conversation has a role adherence failure regardless of whether the medical response was accurate.
Prompt changes are a common trigger for scope drift. A system prompt update intended to improve tone can inadvertently loosen boundaries, and this metric flags that before users do.
Conversation Completeness
Checks whether the agent resolved the user's actual intent by the end of the conversation. Scoring works at the session level. The process pulls each stated intention from the transcript and evaluates whether it was fully addressed, partially addressed, or dropped.
Turn relevancy and completeness measure different things. High turn relevancy confirms each response was on-topic. Completeness tells you whether the underlying request got resolved by the end.
Operational Metrics
Task Completion Rate
The percentage of conversations where the agent completed the user's requested task without transferring to a human. Calculated as successful completions divided by total attempts.
The definition of "successful" needs to be task-specific. For a scheduling agent, it's a confirmed appointment, and for a support bot, it's issue resolution without handoff.
Track this per task type rather than as a single aggregate. A strong overall rate can hide a failing refund flow behind a high-performing booking flow.
Escalation Rate
The percentage of conversations transferred to a human agent. Some of that's by design. High-stakes or out-of-scope issues should reach humans. Trend direction and transfer reason carry the diagnostic value. A spike after a prompt change points directly at a regression. When transfers cluster around a particular intent, that usually points to a gap in the agent's knowledge base or its configured instructions.
Drop-off Rate
The percentage of users who leave the conversation before completing their intent, tracked by turn to identify where they give up.
A drop-off spike at turn 3 on a support flow means something is failing early, whether that's unclear responses, wrong intent recognition, or a dead-end branch. Dissatisfied users tend to exit without submitting feedback, so this metric picks up what post-session scores tend to miss.
CSAT/Sentiment
Post-conversation satisfaction scores and in-conversation sentiment signals. CSAT (Customer Satisfaction Score) measures perceived quality after the conversation ends.
In-conversation sentiment tracking detects frustration patterns turn by turn and points to the exact moment where quality broke down, not a summary score after the fact.
How Cekura Helps With Chatbot Evaluation
Once your chatbot is in production, what you need to measure changes. Individual response quality is still relevant. The additional question at that stage is whether entire conversations hold up across live users, prompt changes, model updates, and edge cases that didn't surface in development.
Cekura handles the chatbot evaluation work that would otherwise require manual review at every stage. Pre-production testing, production monitoring, and regression replay after each change.
Here's how those map to specific features:
- Pre-production simulations: Run automated scenarios before launch to check whether your agent completes appointment booking, support escalation, account verification, and transactional flows with diverse user personas and off-script inputs.
- Custom metrics and LLM-as-a-judge: Define your own scoring criteria in code. Track instruction following, CSAT, hallucinations, tool-call behavior, and conversation completeness against thresholds you set for your use case.
- Production observability: Monitor live conversations for drop-offs, sentiment spikes, latency, and workflow adherence. Replay known problem conversations after prompt, model, or knowledge base changes to confirm regressions are resolved.
- Regression testing in CI/CD: Integrate the full scenario suite into your development pipeline. Any score drop on a given scenario type comes up before it reaches users.
Native integrations work out of the box for Retell, Vapi, ElevenLabs, LiveKit, Pipecat, Bland, and more. The integrations sit on top of your existing stack rather than replacing it.
It's SOC 2-, HIPAA-, and GDPR-compliant for transcript redaction, role-based access, and audit trails.
Frequently Asked Questions
What is the difference between chatbot testing and chatbot evaluation?
Testing validates that specific flows execute correctly. Chatbot evaluation measures response quality across many conversations against defined criteria over time.
It scores knowledge retention, role adherence, and task completion across the full conversation, surfacing failures that flow-level tests tend to miss.
What metrics are used for chatbot evaluation?
The core conversational metrics are turn relevancy, knowledge retention, role adherence, and conversation completeness. Operational metrics include task completion rate, escalation rate, drop-off rate, and CSAT.
Why is single-turn evaluation not enough for chatbot quality?
Single-turn evaluation misses failure modes that only appear across a conversation. Context drift, role boundary violations, and incomplete task resolution only surface when you evaluate the full sequence, not individual responses in isolation.
Can Cekura run a chatbot evaluation automatically?
Yes. Cekura automates pre-production simulations, custom metric scoring, production monitoring, and regression testing in CI/CD for both chat and voice AI agents.
Native integrations are available for Retell, Vapi, ElevenLabs, LiveKit, Pipecat, Bland, and more.