What Is Speaker Diarization? Who Spoke When in Voice AI

The best speaker diarization for a two-party phone call is no diarization at all. Put the caller on the left audio channel and the agent on the right, and "who said that" stops being a machine-learning problem and becomes a routing decision.

This article is about what happens when you lose that luxury, which is more often than you'd expect.

TL;DR

• Speaker diarization labels an audio stream by voice and tells you who spoke when. It never learns anyone's name.
• Published diarization error rates usually exclude overlapping speech and speech near turn boundaries. Those are the exact moments a voice agent lives inside.
• DER is a property of your audio more than your model. One open-source pipeline scores 8.9% on one benchmark and 46.8% on another, running identical code.
• On two-party agent calls, channel separation wins outright. Diarization earns its keep when the channels collapse, during warm transfers, conference bridges, speakerphones, and mono recordings.
• Every speaker-scoped metric you track inherits diarization error. Talk ratio, interruption counts, sentiment, and compliance attribution all fail together.

What Is Speaker Diarization?

Speaker diarization segments an audio recording by voice and labels each segment, answering the question "who spoke when." NVIDIA's NeMo documentation draws the line neatly. Before you diarize, you know what was spoken. You still have no idea who spoke it.

Two jobs happen at once here:

• Speaker detection: counting how many distinct voices exist in the audio.
• Speaker attribution: assigning each stretch of speech to one of those voices.

The output is speaker_0, speaker_1, and timestamps. That's it. No names, no roles.

Diarization vs. the Three Things It Gets Confused With

VAD and endpointing are timing problems, and we went deep on both in our guide to endpointing and turn detection. Diarization is an identity problem without identities.

Speaker recognition is the one that needs an enrolled voiceprint to match against, which is why it carries biometric consent obligations that diarization does not.

How Speaker Diarization Works

Two architectures dominate, and NVIDIA's docs split them as cascaded and end-to-end.

The Cascaded Pipeline

Four stages, four separate models.

1. VAD finds the speech regions.
2. A speaker embedding extractor converts short audio windows into vectors encoding voice characteristics.
3. A clustering module groups those vectors, one cluster per voice in theory.
4. A neural diarizer refines the labels, usually through target-speaker VAD.

The DIHARD III baseline is a useful reference build: x-vectors pulled from 1.5-second windows at 0.25-second steps, scored with PLDA, clustered with agglomerative hierarchical clustering, then resegmented by a variational Bayes HMM.

When you look at that stack, the failure surface announces itself. Sloppy VAD poisons the embeddings, poisoned embeddings poison the clusters, and the clustering step then has to guess how many people are in the room.

End-to-End Neural Diarization

There's one network, where raw audio goes in, and per-frame speaker activity comes out. NVIDIA's Sortformer is the current reference, and its streaming variant keeps a cache of previously heard voices so it can track speakers live instead of waiting for the recording to finish.

End-to-end models handle overlap better, since a frame can carry two active speakers without the architecture objecting. The tradeoff is a hard ceiling on speaker count. NVIDIA's published streaming checkpoints top out at four voices.

The Permutation Problem, and Why Your QA Inherits It

Diarization labels are arbitrary. speaker_0 is not "the caller." It's whichever voice the algorithm happened to index first.

This is a real enough problem that NVIDIA built a model around it. Sortformer resolves permutation by forcing output speakers into arrival-time order, so the first voice heard is always index zero.

Arrival-time ordering helps a lot, but it also carries a hidden assumption. Your agent talks first, which is true for inbound calls and false for every outbound call the instant a human answers with "hello."

So, mapping labels to roles is your job in your code. The diarizer will not do it for you, and it will not warn you when it gets the order backward.

Why Published DER Numbers Overstate Real Performance

Diarization error rate is the standard measure, introduced by the NIST Rich Transcription evaluations that ran from 2002 to 2009. DER sums three failures: missed speech, false alarm speech, and speaker misclassification. Lower is better.

However, the DIHARD III organizers point out that most published work scores DER with speech within 250 ms of reference boundaries removed, and overlapped speech removed entirely.

Their assessment of that convention is blunt. Because short segments and overlapping speech are both common in conversation, the practice "may have resulted in an over-optimistic assessment of performance."

Read that again with a voice agent in mind. The scoring convention deletes turn boundaries and crosstalk, but turn boundaries and crosstalk are roughly the entire job.

The same paper carries an important footnote. When IBM shipped a diarization API in 2017, it worked fine on simple cases, and real user audio exposed it "especially for overlaps, back-channels, and short turns."

What Honest Scoring Looks Like

DIHARD scores with no forgiveness collar and with overlapped speech included, and the numbers move accordingly.

All figures come from the DIHARD III overview paper. Note the pattern in rows one and two, and again in three and four. Handing the system a perfect speech segmentation is worth roughly 7 to 12 points of DER. Your production pipeline does not get handed one.

DIHARD III also reports a second metric, Jaccard error rate, which weights every speaker equally instead of by talk time. The baseline's JER on the core evaluation set is 47.74% against a DER of 20.65%.

That gap is expected behavior, and the scoring tool's own documentation explains why. JER runs higher than DER precisely when one speaker dominates a recording. Worth remembering the next time somebody quotes you a single number.

DER Is a Property of Your Audio

The clearest evidence sits in pyannote's own benchmark table, published in its repository and last updated September 2025. Same open-source pipeline, same code, twelve datasets.

From 8.9% to 46.8%. That's a five-fold spread with nothing changing except the audio going in.

This is why a vendor's headline DER tells you almost nothing about your calls, but it tells you how that model did on somebody else's recordings.

Where Diarization Falls Apart on Real Agent Calls

Overlap and Barge-In

Callers talk over agents. That's the whole point of barge-in support, and it's the single hardest case for single-channel diarization.

Overlapped speech ran at 10.04% of the DIHARD III core development set, and that corpus is mostly recorded conversation rather than the interrupt-heavy traffic a phone agent handles.

Short Turns and Backchannels

"Mm-hm." "Yeah." "Wait, no."

Sub-second acknowledgments carry little voice evidence, so clustering has almost nothing to grip. DIHARD's annotation guidelines split turns on pauses longer than 200 ms, which produces a lot of very small segments.

Those segments are where attribution goes wrong, and they're also where a caller says "no" to a confirmation prompt.

Similar Voices and Warm Transfers

Two voices in the same pitch range are hard to separate. A warm transfer makes it worse by introducing a third voice mid-call, after clustering has already committed to a speaker count.

Streaming Has No Lookahead

Batch diarization reads the whole recording before deciding anything. Streaming diarization commits to a label using only what it has heard so far, then lives with it.

NVIDIA's streaming Sortformer training deliberately limits right-side attention context to 7 frames, about 560 ms, to stop the model leaning on future audio. That constraint is what makes real-time work possible, and it's also why early-call labels are the least stable ones you'll get.

Channel Separation Beats Diarization on Two-Party Calls

Now the practical part.

A phone call between one caller and one agent has a structural advantage that no research benchmark enjoys. You control the recording, record the caller on one channel and the agent on the other, and speaker attribution becomes exact by construction.

There's a DER of zero with no model, no clustering, and no permutation ambiguity.

This is how Cekura's speaker-dependent metrics work. Every call gets dual-channel recording, with high-accuracy VAD run independently per channel, and overlap detected by comparing the two.

Our endpointing guide walks through the mechanics, including the loudness filter that stops a cough from registering as an interruption.

The dependency is explicit in the docs. Talk Ratio, AI Interrupting User, User Interrupting AI, and Stop Time After Interruption each list the same requirement of stereo audio recording with separate channels for each speaker.

So when does diarization actually matter? When the channel guarantee dies.

• Warm transfer to a human. Two voices now share the agent channel.
• Conference bridges and three-way calls. Speaker count exceeds channel count.
• Speakerphone. A spouse, a caregiver, or a colleague joins from across the room, on the caller's channel, uninvited by your architecture.
• Mono recordings from telephony. Some SIP trunks and carrier recordings hand you a single mixed track. DIHARD III's own narrowband material is distributed as monochannel audio, which is exactly the constraint that makes it hard.

Each of those is a normal Tuesday in healthcare and financial services, but none of them announce themselves in your logs.

What Misattribution Costs You Downstream

Diarization is almost never the deliverable. It's the substrate every speaker-scoped signal sits on, which means its errors propagate silently into numbers you trust.

Get the labels wrong, and four signals fail at once:

• Talk ratio inverts. The agent's share of the call gets counted as the caller's.
• Interruption counts flip. The agent talking over the caller reads as the caller talking over the agent.
• Sentiment attaches to the wrong voice. The agent's apology registers as the caller's frustration.
• Compliance attribution collapses. A check asking "did the agent state the disclosure" has no answer, because the system no longer knows which speaker the agent was.

Healthcare makes the stakes concrete, and it accounts for more than half of Cekura's customers.

A 2025 commentary in npj Digital Medicine names speaker attribution errors as a specific safety risk in AI scribes, warning that current systems struggle to consistently distinguish between multiple speakers and can attribute patient statements to clinicians or the reverse.

A patient's reported symptom becomes a clinician's assessment. Same words, different record, different care.

The propagation rate is measurable. A study in Mayo Clinic Proceedings: Digital Health ran fourteen simulated ambulatory encounters through five ambient scribe platforms.

Four numbers from that study:

• 13.9 transcript errors per encounter, averaged across the four platforms that produced a transcript at all.
• 19.5% of those errors carried through into the clinical note.
• 3.0 errors per case carried potential for moderate-to-severe harm on the AHRQ scale.
• 35.8% of correctly reported elements stayed consistent across all five platforms.

The authors flag their own limitation, and it's the interesting bit. Their protocol needs extending to harder circumstances, explicitly including encounters with more than two speakers.

That's the gap. The published evidence covers the easy case.

How to Test Speaker Diarization in Your Pipeline

Every guide on this topic ends with "test on your own audio" and then stops. Here's what you should do instead.

1. Check whether you need diarization at all. Audit your recording path end to end. If every production call arrives as true stereo with a channel guarantee, invest in protecting that guarantee rather than in a diarizer.

2. Find where the guarantee lapses. Trace transfers, conferences, and carrier recording paths. Count what percentage of monthly traffic hits each. That number is your diarization exposure, and it's usually small and expensive.

3. Build ground truth from stereo. Take calls you already have in dual channel, mix them down to mono, and run your diarizer on the mono version. The original channels are your reference labels, free and perfect. No annotation budget required.

4. Score without a collar, with overlap included. Use the dscore conventions DIHARD uses. Anything else flatters the system by hiding the cases you care about. Report DER and JER together.

5. Measure label stability separately. For streaming, count how many times a label flips mid-call. A transcript that settles correctly after three corrections still rendered wrong in your live dashboard.

6. Test the role mapping, not the clustering. Verify that speaker_0 reliably resolves to the agent under outbound conditions, where the human speaks first. This is where arrival-time ordering betrays you.

7. Rerun on every STT change. Diarization behavior shifts with model updates, the same way speech recognition does. Treat a provider version bump as a regression event.

Manual review is not an option here, and there's hard evidence for that claim.

DIHARD's organizers found that annotating who spoke when, to their specification, required highly skilled annotators working with multiple spectrogram displays, at real-time rates typically above 15x and sometimes past 30x.

Many annotators couldn't perform the task at all, even after extensive training.

One hour of audio means fifteen to thirty hours of expert labor. That's the manual baseline you're being asked to scale.

Best Practices for Speaker Attribution

• Protect the channels before you improve the model. Stereo recording is a configuration change. Diarization is an ML project with a permanent error floor. Pick the cheap one.
• Never trust a label as a role. Map speaker_0 to "agent" explicitly, and validate the mapping on outbound calls where a human answers first.
• Give the diarizer a speaker count when you know it. Most APIs accept a hint. A two-party call that you've told the system is two-party is a materially easier problem.
• Score overlap on purpose. If your evaluation excludes crosstalk, it excludes barge-in, and barge-in is a feature you shipped deliberately.
• Watch DER by segment length. Aggregate DER hides the short turns. Bucket your score by utterance duration, and the sub-second bucket will tell you the truth.
• Treat a third voice as a logged event. A speakerphone joiner or a warm transfer changes the compliance picture. Detect it, record it, and route it.

How Cekura Handles Speaker Attribution

Cekura is the testing and observability layer that sits on top of whatever voice stack you already run. Here's what that covers for speaker attribution specifically, in the order you'll need it.

Before you ship (pre-production)

• Simulated calls run against 50+ predefined personalities across accents and communication styles, so attribution gets tested against voices that vary the way real callers do.
• Transcription Accuracy scores the candidate transcript against ground truth generated by two independent transcription models, and reports word error rate alongside it. Our accuracy testing guide covers that layer in full.

Infrastructure

• Dual-channel recording with per-channel VAD, which is what makes Talk Ratio, interruption counts, and Stop Time After Interruption exact rather than estimated.
• Latency reported at P25 through P99 per speaker turn, derived from audio frames instead of transcript timestamps.

Once you're live (observability)

• Voice Change Detection flags unexpected speaker changes inside the agent's own turns, which is the signal that catches a third voice arriving on a channel that should hold one.
• PII redaction applies to transcript and audio together, so speaker-labeled recordings stay handleable in regulated environments.

Your stack stays exactly where it is. Cekura reads natively from Retell, VAPI, ElevenLabs, LiveKit, Pipecat, Bland, and more. It's also SOC 2-, HIPAA-, and GDPR-compliant for transcript redaction, role-based access, and audit trails documented in our trust center.

Speaker diarization is worth solving properly when your calls stop being two-party. Everywhere else, guard the channels and spend the effort on what the labels feed. If you want to see where your own attribution comes apart, book a demo, and we'll run it against real conditions.

Frequently Asked Questions

What is speaker diarization in simple terms?

Speaker diarization is the process of splitting an audio recording into segments and labeling each one by voice, answering "who spoke when."

It counts how many distinct speakers are present and assigns every stretch of speech to one of them, using anonymous labels like speaker_0 rather than real names.

What is the difference between speaker diarization and speaker recognition?

The main difference between speaker diarization and speaker recognition is identity. Diarization separates voices without knowing who they belong to, producing generic labels.

Speaker recognition matches a voice against an enrolled voiceprint to name a specific person, which brings biometric data obligations that diarization avoids.

What is a good diarization error rate?

A good diarization error rate depends entirely on your audio, so no universal target exists. pyannote's open-source pipeline scores 8.9% on clean broadcast audio and 46.8% on egocentric recordings using identical code.

Benchmark your own calls rather than trusting a published figure.

Do voice AI agents need speaker diarization?

No, most two-party voice agents do not need speaker diarization, because dual-channel recording separates the caller and the agent perfectly without any model.

Diarization becomes necessary when multiple voices share one channel, which happens during warm transfers, conference calls, speakerphone conversations, and mono carrier recordings.

Why does diarization struggle with overlapping speech?

Diarization struggles with overlapping speech because clustering assigns each audio segment to a single voice, so simultaneous speakers force a choice.

Most published error rates hide the problem by excluding overlapped speech from scoring, which makes real-world barge-in performance look better than it is.

How do you test speaker diarization without labeled data?

You test speaker diarization without labeled data by mixing existing stereo recordings down to mono and running the diarizer on the mono version.

The original channel assignments serve as perfect ground truth, giving you a free reference for scoring DER on audio that matches your production traffic exactly.