Emotion detection in voice bot calls infers a caller’s state, such as anger or distress, from pitch, loudness, pauses and the transcript. Accuracy drops on real calls: one model scored 63% on acted dialogues and 45.6% on French emergency calls, four emotions each. Cekura tests how voice bots handle emotional callers and scores recorded calls with LLM judge metrics.
TL;DR
- Acoustic emotion detection labels states such as anger or fear from the audio itself, while text sentiment scores only transcript words: Deepgram’s sentiment analysis returns a score from -1 to 1, in English only.
- Real calls scored lower than acted speech in a 2021 study: an end-to-end model reached 63% unweighted accuracy (UA) on IEMOCAP’s acted English dialogues and 45.6% on CEMO’s real French emergency calls to emergency-center agents, four emotions each (arXiv 2110.14957).
- Two classes beat four on real calls: on CEMO’s French emergency calls, one model reached 76.9% UA for anger versus neutral against 45.6% for four emotions, but 61.9% when it trained on acted English dialogues instead (arXiv 2110.14957).
- The EU AI Act has banned emotion inference in workplaces and education institutions since 2 February 2025, lists emotion recognition systems as high-risk under Annex III, whose high-risk rules apply from 2 December 2027 per the European Commission, and requires deployers to tell the people exposed.
- Cekura does not sell an emotion recognition model. Cekura scores how the agent handles emotional callers, with a transcript-based Sentiment metric, a 0 to 100 CSAT score, LLM judge metrics that listen to the recording, and alerts to Slack or Microsoft Teams.
What is emotion detection in voice bot calls?
Emotion detection in voice bot calls is a classification task that gives a caller’s speech a label, such as anger, fear or neutral, or a score for arousal and valence. Acoustic models work from the sound alone: a study of CEMO, a corpus of real French emergency calls, fed 8 kHz Mel spectrograms to a CNN-BiLSTM network with no transcript (arXiv 2110.14957), and its authors name adding the transcript as their next step.
Text sentiment is a related but different measurement: it scores polarity from transcript words. Deepgram’s sentiment analysis scores each word, sentence, utterance and paragraph from -1 to 1. A text score also inherits transcription errors. Per Cekura’s speech-to-text benchmark, AssemblyAI Universal 3.6 Pro, the most accurate of 19 models on 1,000 public clips at 1.77% WER, scored 4.50% WER (95% range 3.66 to 5.34%) on eight recordings from four real phone conversations, though eight recordings is a small sample.
Cekura’s guide to sentiment analysis for voice agent calls covers the transcript side and trigger design.
| Method | What it reads | Typical output | Measured result | Where it breaks |
|---|---|---|---|---|
| Acoustic classifier | Pitch, energy, rate, pauses, spectral shape | Anger, fear, positive, neutral | 45.6% UA, four classes, real emergency calls (arXiv 2110.14957) | Phone channel, unseen speakers, neutral-heavy data |
| Text sentiment | Transcript words | Polarity score from -1 to 1 | English only in Deepgram’s API | Tone that contradicts the words, ASR errors |
| Audio-language model, zero-shot | Recording plus a written prompt | Label with an explanation | Kimi-Audio 60.5% and Qwen2-Audio 55.8% accuracy, averaged over five acted, TV and film datasets, not phone calls (arXiv 2509.25458) | Weak paralinguistic modeling, extra cost per audio minute |
| Text-only LLM, zero-shot | Transcript plus a written prompt | Label with an explanation | 50.9% accuracy, same five datasets | Discards tone entirely |
| Behavioral events | Barge-ins, repeats, silence, requests for a person | Counts and flags | No classifier error, per Cekura’s upsell guide | Lags the emotion, counts “mm-hmm” as a cut-in |
Which acoustic signals reveal frustration, anger and distress on a call?
Acoustic emotion detection is a feature-extraction step that turns pitch, loudness, pace and voice quality into numbers a classifier can label. A 2025 study computed seven such features with the openSMILE toolkit: pitch, speech rate, volume, jitter, shimmer, intensity and articulation rate (arXiv 2509.25458). Higher mean pitch is associated with high-arousal emotions such as anger and fear, and lower pitch with low-arousal states such as boredom, but studies report contradictory results and the features interact rather than act alone (arXiv 2508.04230). Distress can arrive with no words, such as sobbing, which a word-level transcript may miss.
The phone line removes part of the signal. CEMO’s real French emergency calls were sampled at 8 kHz, against 16 kHz for IEMOCAP’s acted English dialogues, so the call audio holds nothing above 4 kHz.
Per Cekura’s Turn Bench, the typical realtime voice model of 13 kept talking through only 6% of filler words like “mm-hmm”, because most models treat any caller sound as an interruption. Turn Bench measures turn-taking, not emotion, but the result shows why a raw interruption count mixes agreement with anger.
How accurate is emotion detection on real phone calls?
Emotion detection accuracy is a measured rate that drops when a model leaves acted speech for real callers on a phone line. A 2021 study tested CEMO, real French emergency calls to emergency-center agents rather than a voice bot: 440 dialogs from 485 callers, after 322 callers labeled neutral throughout were excluded. One end-to-end model scored 63% unweighted accuracy (UA) on IEMOCAP’s acted English dialogues and 45.6% on CEMO, each across four classes (not the same four) where chance is 25% (best run of 5-fold cross-validation; arXiv 2110.14957).
Anger versus neutral reached 76.9% UA on CEMO but fell to 61.9% when the model trained on IEMOCAP, though the authors cannot separate task and language effects in that drop.
Build voice bot logic on a binary negative-versus-neutral flag, which reached 77.5% UA on CEMO. The cost is that one flag cannot tell anger from fear, and it still mislabels more than one speech segment in five.
How do you measure an emotion detection model on your own calls?
An emotion detection evaluation is a labeled test that compares model output with human judgments on your own call audio, using metrics that weight every class equally. Plain accuracy flatters a model on neutral-heavy call audio. In CEMO’s real French emergency calls, 3,382 of 4,825 caller segments (70%) were neutral, so an always-neutral model would score 70% plain accuracy but 25% unweighted accuracy (UA) across four classes (arXiv 2110.14957). Report UA, the mean of per-class recall, and macro-F1.
Human labels set the ceiling. CEMO’s two annotators reached a kappa of 0.61 on caller speech and 0.35 on agent speech. Confusions cluster too: on podcast speech, the third-place system in the Odyssey 2024 challenge labeled contempt as anger 30% of the time on validation data (arXiv 2406.10598).
A four-step measurement checklist:
- Sample calls across accents, line quality and call outcomes.
- Have two people label each segment, and measure their agreement first.
- Score the model, with separate recall for anger and distress.
- Re-run after every model, codec or telephony change.
Each step costs annotator hours, so budget them before buying a model.
How do you test emotion detection in a voice bot before launch?
A pre-launch emotion test is a simulated call that brings a known caller state to the voice bot and checks what the bot does next. Cekura personalities set the simulated caller’s emotional tone, from calm and patient to frustrated or urgent, plus speaking pace, interruption level and background noise (personality docs). Emotion spans 50+ distinct states among the personality dimensions, per Cekura’s guide to voice AI evaluation metrics. A single moment, such as frustration at step four, goes in the evaluator’s instructions instead.
Cekura’s conditional actions can play a roughly 10-second sobbing clip mid-call with a <noise> tag, to check whether the agent notices a caller in distress and asks if they are okay (conditional actions docs). The tag also accepts a direct URL to an audio file, so a recorded clip of an agitated caller can play inside a test call.
Synthetic callers are acted speech, and the personality prompt shapes word choice, not pronunciation. Simulations test the bot’s response logic. Accuracy on real callers still needs labeled production audio.
“The caller who stays polite while their patience drains is the one that breaks an upsell trigger. The words read positive, the state is negative, and a naive gate sells straight into it. Simulate that caller before production does.”
Tarush Agarwal, Co-founder and CEO, Cekura, in Cekura’s guide to sentiment-triggered upsell offers
How do teams use emotion detection on voice bot calls?
Emotion detection on a voice bot call is a routing signal that decides whether the bot keeps the call, changes its approach, or hands off to a person. Common uses are escalation when anger or distress persists, a gentler script for a frustrated caller, and post-call review that surfaces calls worth a human listen.
A false escalation spends a human agent’s minutes on a calm caller, and a miss leaves an upset caller with the bot. Live scoring adds a parallel audio path with extra inference, per Cekura’s guide to sentiment analysis for voice agent calls.
Pair the emotion flag with a deterministic signal, such as a request for a person or a failed task, before it changes control flow. Four-way unweighted accuracy was 45.6% on real French emergency calls (arXiv 2110.14957), so require the state across several turns; the cost is a bot that reacts later. Cekura scores whether the hand-off happened with an LLM judge metric, so a missed escalation fails a test before it becomes a complaint.
Is emotion recognition on customer calls legal under the EU AI Act?
Emotion recognition on customer calls is not banned by the EU AI Act, Regulation (EU) 2024/1689, but it is regulated: the ban covers only workplaces and education, and emotion recognition systems are high-risk and carry a notice duty. Article 3(39) defines these systems as inferring emotions or intentions from biometric data, and Recital 18 excludes merely detecting a raised voice unless used to infer emotion.
- Article 5(1)(f) has banned inferring emotions in workplaces and education since 2 February 2025, except for medical or safety reasons.
- Annex III, point 1(c) lists emotion recognition systems as high-risk. Annex III high-risk rules apply from 2 December 2027 after the AI Omnibus, per the European Commission’s AI Act page and press release.
- Article 50(3), applying since 2 August 2026, requires deployers to inform the people exposed and to process the data under GDPR.
A customer on a support line sits outside the ban but inside the notice duty. Counsel decides whether a transcript-only score counts as biometric.
How does Cekura check emotion handling on voice bot calls?
Cekura is a testing and monitoring platform that scores how a voice agent handles caller emotion, without its own emotion recognition model. Cekura’s pre-defined Sentiment metric reads the transcript and returns positive, neutral or negative, defaulting to neutral when unsure. CSAT averages customer sentiment and agent cooperation into a 0 to 100 score. Both cost 0.2 credits per call (pre-defined metrics docs).
For tone the transcript cannot hold, a Cekura LLM judge metric with Use voice recording turned on listens to the call audio for tone, pace and empathy (LLM judge docs). Audio evaluation adds a per-audio-minute charge, billed on the full call even when a Python metric windows one turn. Open audio-language models struggled with paralinguistic cues in a 2025 study (arXiv 2509.25458), so test the metric on past calls first.
Cekura alerts post to Slack or Microsoft Teams when a metric fails, drifts from its rolling baseline, or crosses a threshold you set, such as CSAT below 70 on 3 of the last 50 calls (alerts docs). Real-time voice agent call monitoring covers setup.
Frequently asked questions
Can a voice bot detect frustration from tone alone?
Yes, within limits. Acoustic models read pitch, energy, speaking rate and pauses without any transcript. One end-to-end model reached 76.9% unweighted accuracy separating anger, a class that includes annoyance and impatience, from neutral on real French emergency calls (arXiv 2110.14957). Four-way labels on the same calls reached only 45.6%. Use a tone-only flag as one input among several, next to explicit requests for a person and failed tasks.
What accuracy should you expect from emotion detection on phone audio?
Expect lower accuracy than on acted speech. In a 2021 study, one model scored 45.6% unweighted accuracy for four emotions on real French emergency calls and 76.9% for anger versus neutral, against 63% and 81.1% on acted English dialogues (arXiv 2110.14957). Trained on the acted dialogues and tested on the real calls, anger versus neutral fell to 61.9%. Measure your own model on labeled calls from your own phone line before you rely on it.
Is emotion detection the same as sentiment analysis?
Not exactly. Sentiment analysis scores polarity, from negative to positive, usually from transcript text: Deepgram’s version scores each word, sentence and utterance from -1 to 1, in English only. Emotion detection assigns categories such as anger, fear or distress, often from the audio itself. Cekura tests both the same way, by driving a known caller state and scoring what the agent does next, as Cekura’s guide to sentiment analysis for voice agent calls explains.
Do you need consent to run emotion detection on calls?
Not under the AI Act’s transparency rule, which asks for notice instead. Article 50(3) requires deployers of an emotion recognition system to inform the people exposed and to process the data under GDPR, which needs a lawful basis. Article 5(1)(f) separately bans inferring staff emotions at work, except for medical or safety reasons. For GDPR, CCPA and call-recording consent, see Cekura’s guide to sentiment analysis for voice agent calls. Counsel decides which rules apply.
Does Cekura have an emotion recognition model?
No. Cekura scores how a voice agent handles emotional callers rather than selling a classifier. Cekura’s pre-defined Sentiment metric reads the transcript, CSAT scores satisfaction from 0 to 100, LLM judge metrics with voice recording turned on listen for tone and pace, and alerts post to Slack or Microsoft Teams when a score fails, drifts or crosses a threshold. To see it on your own calls, book a demo.
