Who said what, and will it be remembered?

Our short paper on persistent speaker attribution was accepted to the IEEE SLT 2026 Demo Track.

Averaged across three settings, our research system's error rate across conversations (SI-cpWER) was 47.1%, against 54.8% for the next system, about 14% fewer errors. The technical report calls this system ThyVoice. In the charts and tables below it appears as Thine.

Every main table from the report is in Full results at the end, each with a short note on how to read it.

Why who said what has to hold up

AI notetakers and ambient assistants are only as good as their record of who said what. Summaries, action items, search and answers are all built on that record.

Maya says, “I'll send the revised budget on Friday.” In a later conversation she says, “The budget is ready, I've sent it.” Both transcripts can get her words exactly right. If the system stores Maya as two different people, though, the promise stays open, and a search for her outstanding commitments finds the promise without its resolution.

That is the problem Thine is built around. Thine learns from the in-person conversations of your day and keeps track of the open loops in them, and every open loop belongs to someone.

Standard accuracy metrics can't see this failure. Word accuracy (WER) ignores speakers entirely. Per-conversation scoring (cpWER) matches speakers afresh in every recording, so “Speaker 1” on Monday and “Speaker 1” on Friday never have to be the same person. We evaluated that directly, by checking whether each person keeps one identity across every recording (SI-cpWER).

Everyday conditions make this harder. People sit at different distances, rooms are noisy, people talk over each other, and the same person comes back later in a different room, on a different device, through a different microphone.

How we evaluated

We benchmarked eight speech systems. Five are commercial pipelines built on OpenAI (GPT-4o Transcribe Diarize), ElevenLabs (Scribe v2), Deepgram (Nova-3), AssemblyAI (Universal-3 Pro) and pyannoteAI (Precision-2). Two are open-source research systems, SE-DiCoW and WhisperX. The eighth is our research system.

Each comparison system produces a transcript with its own speaker labels for every recording. To link speakers across recordings, all seven use the same pyannoteAI voice-matching service; our research system uses its own. These results describe the configured systems above. They are not scores for each vendor's own identity product or for finished apps.

We used three settings:

• Office meetings. All 129 meetings in the NOTSOFAR evaluation set: 12 people, about 13 hours of audio, and roughly 30% overlapping speech. Each meeting is one recording from one microphone.
• The same meetings with added noise. Room noise, background chatter and short sounds such as keyboards and doors, mixed in at about 10 dB signal-to-noise on average.
• Dinner parties. Two real dinner-party recordings from CHiME-6, captured on far-field devices in homes, with eight people in total. Each recording is more than two hours long, so systems process it in roughly five-minute pieces, one after another.

Every setting starts with no known voices, and recordings arrive one after another in a fixed order. Nobody enrolls their voice in advance, so each system has to decide on its own when it is hearing someone new and when it is hearing someone again.

What we found

Strong results within one conversation did not carry over

Once the same person had to be recognized across conversations, our research system's error barely changed. Every other system's error rose, several of them sharply, including the system with the best word accuracy and the fewest errors within a conversation (cpWER).

Lower error than every commercial system in every setting

Across office meetings, the same meetings with added noise, and real dinner parties, our research system had lower error than every commercial system. The open-source SE-DiCoW led both office settings. At the dinner parties, our research system led all eight by 10 points.

The fewest words put in someone's mouth

A transcript can go wrong in two ways. It can say something that wasn't said: a wrong word, an extra word, or a real word given to the wrong person. Or it can leave words out.

Our research system made the fewest errors of the first kind (substitutions plus insertions), averaged across the three settings, and the fewest extra words in every setting. The trade-off is that it left out more words than the ElevenLabs, SE-DiCoW and OpenAI systems.

Overlapping audio is bad evidence for recognizing someone

When people talk over each other, the mixed audio can still be transcribed, but it is the wrong audio to trust when deciding who someone is.

We evaluated four ways of handling overlap on 20 office meetings. Two of them keep the overlap for transcribing and differ only in the audio used to recognize voices, clean or overlapping. Their error within a conversation is nearly identical (42.5 vs 42.9 cpWER), but recognizing from overlapping audio raised the error across conversations by more than half, from 41.4 to 67.8 (SI-cpWER). Separating overlapping voices first scored best.

This is how our research system handles overlap. It works out who is speaking when, separates two overlapping voices only where it can tell them apart with confidence, transcribes each person's audio on its own, and sets a higher bar for updating someone's voice profile than for recognizing them.

Full results

This section has every main table from the report. All numbers are error rates in percent, where lower is better, unless a column says otherwise. Mean is the simple average of the three settings. Clean is the office meetings, Noisy is the same meetings with added noise, and Dinner is CHiME-6. Thine is our research system. ElevenLabs results on the office settings are the average of three full runs. Per-setting breakdowns and the appendices are in the technical report.

What the metrics measure

• WER (word error rate): wrong, missing and extra words, ignoring who said them.
• cpWER: the same errors, but each person's words are scored separately, with speakers matched afresh in every recording.
• SI-cpWER: cpWER with one speaker mapping held across every recording, so each person has to keep the same identity throughout.
• Substitutions, deletions and insertions: wrong words, missing words and extra words. In cpWER, a real word given to the wrong person also counts here.
• DER and JER: how well a system tracks who is speaking when, before any words are involved.
• Purity and coverage: whether each speaker group a system creates holds one person (purity), and how much single-speaker speech gets any speaker label at all (coverage).

Errors across conversations (SI-cpWER)

Our research system has the lowest mean and the lowest error of the commercial systems in every setting. SE-DiCoW has the lowest error on the two office settings.

Errors within a conversation (cpWER)

The last column is how much each system's mean changes from cpWER to SI-cpWER. It is the net effect of linking people across recordings, which can add errors or fix speakers that were split within a recording.

How many people each system thinks there are

These are the identities each system ends up with after all recordings, against the true number of people. A count close to the true number is a good sign, but it does not prove the right words went to the right person. For the comparison systems, the shared pyannoteAI service did the matching. ElevenLabs' office counts are a range across three runs, and unknown-speaker labels are not counted.

What kind of errors each system makes

Means across the three settings, from the within-conversation scoring (cpWER). Substitutions plus insertions are words a transcript asserts wrongly: wrong words, extra words, and real words given to the wrong person. Deletions are words it leaves out. The report breaks these down by setting and by error type.

Word accuracy (WER)

ElevenLabs had the best word accuracy in every setting. Two rows are recognizers run on the original mixed audio: Parakeet is the recognizer in pyannoteAI's pipeline, and Qwen3-ASR is the recognizer inside our research system. That Qwen3-ASR number is not our system's end-to-end word accuracy, because our system transcribes each person's own audio rather than the mix.

Who speaks when (DER and JER)

The DiariZen md-v2 component in our research system had the lowest DER and JER in every setting. Outputs differ: some systems return a native timeline of who speaks when, OpenAI returns segments, and ElevenLabs, AssemblyAI and Deepgram label each word with one speaker. For those three we rebuilt speaker turns from the word labels, joining gaps of up to 0.25 seconds, so their scores partly reflect that rebuild.

Speaker evidence before voice profiles are made

Office settings only, shown as Clean / Noisy. These numbers describe the material a system could use to build voice profiles, not the quality of the profiles it ends up storing. Purity and coverage need to be read together, since high purity alone can hide speech that never got a label. Overlap found is the share of overlapping speech a system marks as overlapping. For outputs that label each word with one speaker, 0% means the output has no way to show two people at once; it says nothing about what the provider detects internally.

The overlap experiment

Twenty office meetings, using our research system.

How we set up voice profiles for the comparison systems

Errors across conversations (SI-cpWER) on 20 office meetings, one run each. When a comparison system meets a new speaker, the shared pyannoteAI service builds a voice profile from that speaker's audio. We compared building it from the speaker's single longest turn against joining all of their segments together (pooled). Pooling cut errors for every commercial system in the clean setting, by 4.9 to 13.8 points, and for four of the five in the noisy setting, by 15 to 26 points. pyannoteAI was the exception there. Our main results give every comparison system the pooled style of setup, up to 30 seconds of each speaker's audio joined together, rather than a single turn.

Scope

• English conversations, one microphone per recording.
• Two datasets in three settings. The dinner-party setting is two long recordings, and the noisy setting adds noise to the office recordings rather than capturing new audio.
• Recognition across conversations is evaluated by processing recordings in one fixed order, not by following the same people over weeks.
• We evaluated the speech technology inside these systems, not complete products.

Make this layer visible

This is the layer every AI notetaker and ambient assistant is built on. It should be reported as openly and comparably as word accuracy is today. We evaluated the speech technology rather than finished products, and we hold the Thine app to the same standard.

The full methodology is in the technical report, which expands on the accepted short paper.

We're continuing to improve this layer, and we'll keep sharing what we learn.