What is intent recognition? How it works and how to test it on calls

Intent recognition is how a conversational agent works out what a user wants, such as booking a visit or disputing a charge, and maps it to an action. In a voice agent it runs on a transcript. Its accuracy depends on speech recognition, on how alike your intents are, and on how it handles requests it cannot serve.

That last part is where a single accuracy number can mislead. A classifier can look accurate on the intents it knows and still miss most of the requests it was never built for. This guide covers how intent recognition works, the metrics that expose those gaps, and a test plan you can run on real calls.

TL;DR: Intent recognition

• Intent recognition, also called intent detection or intent classification, assigns each user turn to one of a fixed set of goals, or to a fallback when none fits. Entities such as dates and account numbers are extracted alongside it.
• Overall accuracy hides the failures that matter. On the CLINC150 benchmark, BERT reached 96.9% in-scope accuracy but caught only 40.3% of out-of-scope queries when out-of-scope was trained as one extra class.
• Speech recognition errors lower intent accuracy, but not one for one. On the SLURP spoken dataset, intent accuracy fell from 84.84% on human transcripts to 73.41% on ASR output with about 24% word error rate.
• LLM voice agents usually have no intent label to score. You test intent through outcomes: which tool the agent called, which route it took, and whether it kept both needs in a two-part request.

What is intent recognition?

Intent recognition is the task of classifying a user’s utterance by its goal. “I lost my card,” “my card is gone,” and “someone took my wallet with the card in it” all map to one intent, such as report_lost_card. The agent then runs the flow, tool call or handoff tied to that intent.

Papers and vendor docs also call the same task detection or classification. In chatbot intent recognition the input is typed text, while a voice agent works from a transcript of speech.

Keyword matching misses what the user means. “Cancel my appointment” and “don’t cancel my appointment” share three words and mean opposite things, while “push my visit back” and “reschedule my appointment” share only “my” and mean the same thing.

The step sits between speech recognition and dialogue management in the four layers of a voice assistant pipeline. It usually travels with entity extraction, also called slot filling. The intent says what the user wants, and the entities carry the details needed to act.

The last row matters as much as the first two. A production agent hears many requests it was never built for, and recognizing those correctly is part of the job.

How intent recognition works in a classic NLU pipeline

A classic NLU pipeline treats intent recognition as supervised text classification. You define a list of intents, collect example phrases for each, and train a model that scores every new utterance against every intent.

The steps look much the same across platforms:

1. Define intents. Each intent is a goal your agent can act on, such as check_balance or transfer_to_agent.
2. Add training phrases. Each intent gets example utterances, ideally drawn from real user logs rather than invented at a desk.
3. Train the classifier. Older systems used SVMs or CNNs, and newer ones fine-tune transformer encoders such as BERT.
4. Score and threshold. At runtime the model returns its best intent with a confidence score, and low scores go to a fallback.

Amazon Lex V2 shows step four clearly. Per the Lex V2 guide to intent confidence scores, each response carries the top intent plus up to four alternatives with scores. If every intent scores below the confidence threshold, Lex returns AMAZON.FallbackIntent.

The same guide warns that the scores are comparative and may change as Lex improves its models. Two close scores, such as 0.75 and 0.72, signal ambiguity between two intents rather than a confident answer. When the top two scores are close, confirm instead of guessing: your Lambda function can pass an alternative intent to a ConfirmIntent dialog action.

The amount of training data shows up in the score. In the BANKING77 intent detection paper, one detector (USE+ConveRT) reached 93.36% accuracy with the full training set but 85.19% with 10 examples per intent. The cost is labeling time, since every extra example is human work.

How many intents should an agent have?

An intent set is the list of goals an agent can recognize, and its size is a design tradeoff. Too few intents hide differences that need different actions: folding “reschedule” into “book” creates duplicate bookings. Too many near-duplicates confuse the classifier, as in BANKING77, where the authors note overlap between intents such as reverted top-up and failed top-up. Split two intents only when they trigger different actions. Cost: every intent needs real training phrases and test utterances to maintain.

How LLM-based voice agents handle intent

An LLM-based voice agent recognizes intent implicitly. There is usually no list of trained intents. The model reads the transcript, the system prompt and the tool definitions, then picks a reply or tool call that fits the request.

That design changes what intent means in practice. The intent stops being a label the system emits and becomes the action the agent takes. Calling cancel_appointment is the intent decision, and a polite reply that never calls the tool is a miss.

With AI intent recognition inside an LLM, new intents are cheap to add. You describe a capability in the prompt or add a tool, with no retraining. The model also copes with paraphrase and two-part requests without hand-written examples.

The tradeoff is domain knowledge. A 2023 EMNLP evaluation of ChatGPT on open-world intent discovery found that ChatGPT beat a non-fine-tuned BERT with no labeled examples but trailed fine-tuned models on banking and CLINC data. The authors name missing domain knowledge as the main reason for the gap on generalized intent discovery. That study tested an earlier model, so check the direction on your own data.

Which metrics measure intent recognition accuracy?

Intent recognition accuracy is the share of test utterances assigned to the correct intent. It is easy to report and easy to misread, because it blends strong and weak intents into one number. For session-level measures such as escalation rate and drop-off rate, see the guide to chatbot evaluation methods and metrics.

Per-intent precision and recall show where the errors go. For any intent X:

• Precision = utterances correctly labeled X ÷ all utterances labeled X
• Recall = utterances correctly labeled X ÷ all utterances that truly are X
• F1 = 2 × precision × recall ÷ (precision + recall)

Macro F1 averages F1 across intents equally, so a rare intent counts as much as a common one. Micro averaging weights by volume, and for single-label classification it equals accuracy. Report macro F1 when rare intents carry the most risk, such as fraud reports or cancellations.

Entities need their own score. For each slot, such as a date or account number, count the share of test utterances where the agent captured the exact value. A wrong digit fails the call even when the intent is right. For spoken input, the SLURP authors proposed SLU-F1 because strict span matching over-penalizes misalignments that ASR errors cause.

Worked example: three appointment intents

Say you run 300 test calls through a scheduling agent, 100 per intent. The numbers below are illustrative. Rows are the true intent, and columns are what the agent did.

Overall accuracy is 268 ÷ 300, or 89.3%. That looks fine until you read the rows.

Fifteen reschedule requests became new bookings. Each one keeps the old slot and adds a duplicate, so most of the business cost sits in one cell. The fix is targeted: add reschedule phrasings that mention an existing booking, or confirm when both intents score close.

Confusion between similar intents

Narrow domains are harder because their intents sit close together. The BANKING77 dataset, 13,083 banking queries over 77 intents, was built to test exactly this. Its authors note that some intents partially overlap, such as “card lost” and “link to existing card,” or “reverted top-up” and “failed top-up.”

In the same paper, the USE+ConveRT detector scored 97.16% on the 10-domain CLINC150 set but 93.36% on single-domain BANKING77, both with full training data. If your agent serves one business, plan around the harder number.

What is out-of-scope detection in intent recognition?

Out-of-scope detection is an agent’s ability to recognize a request it does not support and send it to a fallback. The CLINC150 out-of-scope intent paper built a dataset for it: 23,700 queries, with 22,500 covering 150 intents across 10 domains and 1,200 out of scope.

The results show why accuracy alone misleads. With the full training set and out-of-scope trained as one extra class, BERT scored 96.9% in-scope accuracy but recognized only 40.3% of out-of-scope queries. A probability threshold did better for BERT, at 52.3% recall and 96.2% in-scope accuracy, and even with extra out-of-scope training examples the best system reached 66% out-of-scope recall.

The authors chose recall on purpose. An out-of-scope query treated as in scope produces a completely wrong response. A false fallback costs less, because the agent asks the user to try again.

Out-of-scope recall = out-of-scope test utterances sent to fallback ÷ all out-of-scope test utterances

Track it next to the false fallback rate, the share of in-scope utterances wrongly sent to fallback. Raising the confidence threshold improves the first and worsens the second. Set the threshold on a labeled test set, and accept that each point of recall costs some in-scope users a retry.

How ASR errors change intent accuracy on calls

On a phone call, the agent never sees what the caller said. It sees what speech recognition wrote down, so every misheard word that changes meaning flows into intent and entity decisions.

The SLURP spoken language understanding dataset holds about 72,000 recordings of single-turn home-assistant commands across 18 scenarios. Intent there is a scenario plus an action. The authors scored the same NLU model, HerMiT, on human transcripts and on ASR output.

The table shows two things. ASR errors cost real accuracy, about 11 points at 24% WER. But intent accuracy fell less than WER rose. The authors hypothesize that resilient scenario and action encodings explain this, and many word errors likely land on words that do not change the goal.

The words that break count for more than the error rate. A dropped “not,” a misheard verb or a wrong product name can flip an intent, while a missed “the” rarely does. That is why WER alone does not tell you how many intents will fail, and why voice agent accuracy testing scores transcription, intent and task completion as separate layers.

Real calls are also harder than short public clips. Per Cekura’s Converse-STT benchmark, last updated 16 September 2026, 16 of 19 speech-to-text models scored a higher WER on eight licensed phone-call recordings than on 1,000 short public clips, and on the recordings WER ranged from 2.93% to 12.45%. AssemblyAI Universal 3.6 Pro, for example, scored 1.77% on the clips and 4.50% on the calls. The call set is only eight recordings from four phone conversations, and 14 models sit within the leader’s margin of error there.

How to test intent recognition before launch

Testing intent recognition means running realistic utterances through the full agent, voice included, and scoring the action it takes against what the user wanted. Six steps cover the main failure modes.

1. Build a labeled utterance set from real logs. Pull real phrasings per intent from transcripts or chat history and label each by hand. Cost: labeling time grows with every intent.
2. Write near-miss pairs. For each pair of intents that share words, write utterances that differ by one detail, such as “move my appointment” and “make an appointment.” Cost: someone who knows the business rules has to write them.
3. Add an out-of-scope set. Collect requests your agent should decline or escalate, styled like real traffic, and measure out-of-scope recall. Cost: the CLINC150 authors found only about 69% of queries collected as out of scope actually were, so plan for cleanup.
4. Speak the tests as well as typing them. Run the same utterances as audio with different accents, background noise and speaking pace. Cost: voice runs take longer and are billed per minute, while chat runs are billed per reply and cost less.
5. Include multi-intent and intent-shift turns. Script callers who state two needs at once, change their goal mid-call, or bury the real request in a long story. Cost: multi-turn scenarios are slower to write and to judge.
6. Score per intent and rerun on every change. Report the confusion matrix, per-intent recall and out-of-scope recall after each prompt, model or ASR change. Cost: a full rerun on every release, which is why the suite should stay focused on high-risk intents.

What changes when an LLM agent replaces an intent classifier

Replacing an intent classifier with an LLM moves intent testing from labels to outcomes. You no longer check whether the model emitted check_balance. You check whether it called the balance tool, with the right account, before it answered.

Three practical changes follow:

• Score actions, not labels. Assert on tool calls and routes, because a transcript can sound right while the tool never ran.
• Repeat every scenario. The same scenario can pass once and fail the next time. Cekura’s benchmarks report pass³, which counts a scenario only when all three repeated runs pass.
• Treat prompt edits like model changes. A new instruction for one intent can shift behavior on others, so rerun the full intent suite after each edit.

Two-part requests get easier and riskier at once. An LLM understands “I have a billing question, and I also want to shop” with no special training. It can also answer the first need and forget the second.

Cekura’s Medicare workflow study (Study 02, July 2026) shows one way to test this. It ran one byte-identical Medicare insurance voice agent on six voice platforms, with 23 evaluators repeated three times. Seven evaluators target call reason. They include a caller with two needs, a caller who changes purpose mid-call, and one who buries the real ask in a long story.

Workflow pass³ combines Expected Outcome and Mock Tool Accuracy, and an evaluator passes only if all three runs pass. It ranged from 95.7% on Retell to 65.2% on Vapi. Those figures cover the whole workflow, not intent alone, and come from one agent in one domain. The study is reported separately from the main platform leaderboard.

How Cekura tests intent recognition on voice and chat agents

Cekura tests intent recognition by simulating callers, scoring the actions the agent takes, and separating speech errors from understanding errors. The aim is to show which layer broke when a call fails.

• Cekura personalities set the simulated caller’s voice, accent, pace and background noise. Per the accent testing guide, Override Personality runs the same evaluators against every selected accent, so only the voice changes.
• Cekura’s Transcription Accuracy metric compares the agent’s transcript of the simulated caller with what the caller actually said. Names, nouns, numbers and negation flips carry the most weight, so a low score points to ASR rather than intent logic.
• Cekura’s expected-outcome judge reads the full transcript, including tool calls. It fails an evaluator when the agent never called the required tool or passed the wrong arguments.
• In production, Cekura’s Enum metric labels each call with one of your predefined categories. A primary-intent Enum metric turns live calls into a labeled stream you can compare with what the agent did.
• Cekura’s Deep Research audits a window of production calls for failure modes no metric covers, and returns each one with a suggested fix and example calls.

For scenario-level checks on phrasing variation, intent drift and confusion between similar intents, see the guide to conversation-level intent accuracy testing.

There are limits. In simulations, Transcription Accuracy scores only the simulated caller’s turns, because there is no independent ground truth for the agent’s own speech. On production call logs it costs 1.0 credit per minute, so metric sampling on a share of calls keeps spend down. Deep Research is enabled per project through support, and LLM-judged metrics still need spot checks against human labels.

Where to start with your own intent tests

Pull 50 recent calls from the two intents that sound most alike and label them by hand. Compare your labels with what the agent did, then build a small confusion matrix. The largest off-diagonal cell is your first fix.

Then add 20 out-of-scope requests and count how many reach a fallback. If that recall is low, you have found the failure your accuracy number was hiding. Treat the first run as a smoke test: 14 of 20 caught is 70% recall, but the 95% Wilson interval runs from about 48% to 85%, so grow the set before you fix a release target.

Frequently asked questions

What is the difference between intent recognition and intent detection?

There is no practical difference, because both terms describe mapping a user’s utterance to a goal. Research papers and vendor docs use intent recognition, intent detection and intent classification interchangeably. Some teams use “detection” more narrowly, for deciding whether any supported intent is present at all.

What is a good intent recognition accuracy?

A good accuracy figure depends on how many intents you have and how alike they are. With full training data, the best models in the BANKING77 paper reached about 94% on BANKING77 and 97% on CLINC150. Out-of-scope recall on CLINC150 peaked at 66%, so set per-intent recall targets instead of one overall number.

How do you improve intent recognition accuracy?

Improve accuracy by fixing the largest confusion first. Find the biggest off-diagonal cell in your confusion matrix, add real phrasings for that pair, merge intents users do not distinguish, and add out-of-scope examples. More data helps: in the BANKING77 study, one detector rose from 85.19% accuracy at 10 examples per intent to 93.36% on the full set. Retest on speech, because ASR errors add failures that text tests miss.

Can an LLM replace an intent classifier?

An LLM can replace an intent classifier in many agents, especially when requests vary widely or come in several parts. The cost is control: an LLM does not return a calibrated intent score by default, so you set the confirm-or-fall-back rule in the prompt and test it, and the same input can produce different actions across runs. Consider keeping a classifier or a hybrid when you need fixed, auditable routing for a small set of intents.

How do you handle multiple intents in one utterance?

Handle multiple intents by detecting each one, confirming the order with the user, and tracking each until it is resolved. Classic classifiers often return one label per turn, so multi-intent support needs extra design. LLM agents accept both needs naturally, but test that the second request survives after the first is served.

How does Cekura test intent recognition?

Cekura simulates callers with varied phrasing, accents and noise, then scores whether the agent took the right action. Its Transcription Accuracy metric separates speech errors from understanding errors, and expected-outcome checks verify tool calls. In production, Enum metrics and Deep Research surface new intent failures.

To see how your agent handles accents, noise and two-part requests, book a Cekura demo or sign up and start with free credits.