Does it spell the names you taught it, and leave the rest alone?
On audiobooks, Detail Keeper spells taught names as well as ElevenLabs and better than Deepgram; ElevenLabs changes slightly fewer other words. In conversations, ElevenLabs is ahead in 2 of 8 conditions, Deepgram is behind ours in 6 of 8.
Ours is taught each name with one recording of it; the services get only the spelling. Both sets are sealed: held out from building and choosing our model, and scored once (own-names-bench v1.3.0).
Names right
Percent of taught names written with exactly the taught letters. Higher is better. The thin line, and the brackets, are the 95% interval.
Audiobooks 1,100 mentions
Conversations 2,327 mentions, 8 conditions (by condition)
Totals over the 8 conditions have no interval, so there is no single verdict: each condition gets its own, counted here.
Other words it got wrong
Lower is better.
| Detail Keeper | ElevenLabs | Deepgram | |
|---|---|---|---|
| Turned an untaught name into a taught onethe name was off the list; another listed name was written instead | 3.6% [1.1, 6.6]40 of 1,100 | 1.5% [0.5, 2.4]16 of 1,100ours worse | 5.9% [3.9, 7.6]65 of 1,100no measurable difference |
| Added a name where none was saidin name-free clips | 0.2% [0.0, 0.4]2 of 1,200 | 0.2% [0.0, 0.4]2 of 1,200no measurable difference | 1.3% [0.4, 2.7]16 of 1,200ours better |
| Ordinary-word errorthe words outside the names | 2.1% [1.7, 2.6] | 1.5% [1.2, 1.9]ours worse | 3.1% [2.4, 3.8]ours better |
| Detail Keeper | ElevenLabs | Deepgram | |
|---|---|---|---|
| Turned an untaught name into a taught onethe name was off the list; another listed name was written instead | 4.0%93 of 2,327 | 8.8%204 of 2,327ours better: 5 of 8no measurable difference: 3 of 8 | 8.2%190 of 2,327ours better: 4 of 8no measurable difference: 4 of 8 |
| Added a name where none was saidin name-free clips | 0.5%6 of 1,196 | 2.4%29 of 1,196ours better: 1 of 6no measurable difference: 5 of 6 | 0.3%3 of 1,196no measurable difference: 6 of 6 |
| Ordinary-word errorthe words outside the names | 11.2% | 10.3%ours better: 1 of 8ours worse: 3 of 8no measurable difference: 4 of 8 | 17.9%ours better: 8 of 8 |
By condition (8)
Eight conditions, built from the test splits of VoxPopuli (European Parliament speech) and LibriTTS-R (read speech), with new noise from FSD50K and simulated rooms: 2,327 mentions and 1,196 name-free pieces (102.9 minutes), 40 names per list. Each name is taught from one recording of it: in all but one condition, the same speaker for 1,359 of 1,979 mentions and another person for the rest; in the condition taught by another person, always another person. Each bar has its 95% interval; the mark beside a service is its verdict against ours on the same items.
- Detail Keeper
- ElevenLabs Scribe v2
- Deepgram Nova-3
- ours better
- ours worse
- no measurable difference
Bar: percent. Line: 95% interval.
Names right
higher is better · 0–100%
Parliament speech, names right: Detail Keeper 98.5%, 130 of 132 (95% interval 96.2 to 100.0); ElevenLabs Scribe v2 97.7%, 129 of 132 (95% interval 94.8 to 100.0): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 +0.8 points (−2.3 to +4.2); Deepgram Nova-3 96.2%, 127 of 132 (95% interval 92.5 to 99.2): no measurable difference, Detail Keeper minus Deepgram Nova-3 +2.3 points (−1.5 to +6.5).
Phone line, names right: Detail Keeper 98.1%, 265 of 270 (95% interval 96.3 to 99.6); ElevenLabs Scribe v2 97.0%, 262 of 270 (95% interval 94.9 to 98.9): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 +1.1 points (−1.1 to +3.4); Deepgram Nova-3 90.4%, 244 of 270 (95% interval 86.7 to 93.6): ours better, Detail Keeper minus Deepgram Nova-3 +7.8 points (+4.5 to +11.4).
Far from the microphone, names right: Detail Keeper 82.9%, 276 of 333 (95% interval 78.3 to 87.2); ElevenLabs Scribe v2 88.0%, 293 of 333 (95% interval 84.4 to 91.5): ours worse, Detail Keeper minus ElevenLabs Scribe v2 −5.1 points (−9.1 to −1.4); Deepgram Nova-3 60.7%, 202 of 333 (95% interval 54.7 to 66.7): ours better, Detail Keeper minus Deepgram Nova-3 +22.2 points (+17.2 to +27.3).
Street noise, names right: Detail Keeper 93.8%, 300 of 320 (95% interval 90.9 to 96.4); ElevenLabs Scribe v2 95.6%, 306 of 320 (95% interval 93.3 to 97.7): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −1.9 points (−4.9 to +1.2); Deepgram Nova-3 85.3%, 273 of 320 (95% interval 81.4 to 89.2): ours better, Detail Keeper minus Deepgram Nova-3 +8.4 points (+5.0 to +12.0).
Very quiet, taught up close, names right: Detail Keeper 78.7%, 236 of 300 (95% interval 73.7 to 83.5); ElevenLabs Scribe v2 84.3%, 253 of 300 (95% interval 79.8 to 88.7): ours worse, Detail Keeper minus ElevenLabs Scribe v2 −5.7 points (−9.9 to −1.4); Deepgram Nova-3 54.0%, 162 of 300 (95% interval 48.0 to 59.9): ours better, Detail Keeper minus Deepgram Nova-3 +24.7 points (+19.1 to +30.2).
A sound-alike on the list, names right: Detail Keeper 92.1%, 258 of 280 (95% interval 88.9 to 95.1); ElevenLabs Scribe v2 92.5%, 259 of 280 (95% interval 89.3 to 95.4): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −0.4 points (−3.9 to +3.2); Deepgram Nova-3 85.4%, 239 of 280 (95% interval 80.7 to 89.7): ours better, Detail Keeper minus Deepgram Nova-3 +6.8 points (+2.6 to +11.2).
Short snippets, names right: Detail Keeper 85.2%, 293 of 344 (95% interval 81.1 to 89.0); ElevenLabs Scribe v2 84.6%, 291 of 344 (95% interval 80.2 to 88.8): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 +0.6 points (−4.2 to +5.4); Deepgram Nova-3 62.2%, 214 of 344 (95% interval 56.5 to 67.6): ours better, Detail Keeper minus Deepgram Nova-3 +23.0 points (+17.7 to +28.5).
Taught by another person, names right: Detail Keeper 99.1%, 345 of 348 (95% interval 98.0 to 100.0); ElevenLabs Scribe v2 99.4%, 346 of 348 (95% interval 98.5 to 100.0): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −0.3 points (−1.5 to +0.9); Deepgram Nova-3 97.7%, 340 of 348 (95% interval 96.1 to 99.1): no measurable difference, Detail Keeper minus Deepgram Nova-3 +1.4 points (−0.3 to +3.3).
Untaught overwritten
lower is better · 0–20% · count
Parliament speech, untaught overwritten: Detail Keeper 0.0%, 0 of 132 (95% interval 0.0 to 0.0); ElevenLabs Scribe v2 0.8%, 1 of 132 (95% interval 0.0 to 2.4): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −0.8 points (−2.4 to 0.0); Deepgram Nova-3 2.3%, 3 of 132 (95% interval 0.0 to 5.1): no measurable difference, Detail Keeper minus Deepgram Nova-3 −2.3 points (−5.1 to 0.0).
Phone line, untaught overwritten: Detail Keeper 2.6%, 7 of 270 (95% interval 0.8 to 4.7); ElevenLabs Scribe v2 8.1%, 22 of 270 (95% interval 5.1 to 11.4): ours better, Detail Keeper minus ElevenLabs Scribe v2 −5.6 points (−8.8 to −2.4); Deepgram Nova-3 9.3%, 25 of 270 (95% interval 5.9 to 12.8): ours better, Detail Keeper minus Deepgram Nova-3 −6.7 points (−10.1 to −3.3).
Far from the microphone, untaught overwritten: Detail Keeper 5.4%, 18 of 333 (95% interval 3.1 to 7.9); ElevenLabs Scribe v2 11.4%, 38 of 333 (95% interval 8.3 to 14.8): ours better, Detail Keeper minus ElevenLabs Scribe v2 −6.0 points (−9.4 to −2.7); Deepgram Nova-3 9.9%, 33 of 333 (95% interval 6.9 to 13.3): ours better, Detail Keeper minus Deepgram Nova-3 −4.5 points (−8.2 to −0.9).
Street noise, untaught overwritten: Detail Keeper 5.6%, 18 of 320 (95% interval 3.3 to 8.3); ElevenLabs Scribe v2 6.9%, 22 of 320 (95% interval 4.3 to 9.6): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −1.2 points (−4.0 to +1.6); Deepgram Nova-3 10.6%, 34 of 320 (95% interval 7.4 to 13.9): ours better, Detail Keeper minus Deepgram Nova-3 −5.0 points (−8.3 to −1.8).
Very quiet, taught up close, untaught overwritten: Detail Keeper 5.3%, 16 of 300 (95% interval 3.0 to 7.9); ElevenLabs Scribe v2 11.0%, 33 of 300 (95% interval 7.8 to 14.4): ours better, Detail Keeper minus ElevenLabs Scribe v2 −5.7 points (−8.9 to −2.5); Deepgram Nova-3 8.3%, 25 of 300 (95% interval 5.3 to 11.7): no measurable difference, Detail Keeper minus Deepgram Nova-3 −3.0 points (−6.7 to +0.6).
A sound-alike on the list, untaught overwritten: Detail Keeper 5.4%, 15 of 280 (95% interval 2.8 to 8.2); ElevenLabs Scribe v2 12.5%, 35 of 280 (95% interval 8.6 to 16.9): ours better, Detail Keeper minus ElevenLabs Scribe v2 −7.1 points (−11.9 to −2.8); Deepgram Nova-3 15.4%, 43 of 280 (95% interval 11.4 to 19.5): ours better, Detail Keeper minus Deepgram Nova-3 −10.0 points (−14.0 to −6.1).
Short snippets, untaught overwritten: Detail Keeper 4.9%, 17 of 344 (95% interval 2.8 to 7.3); ElevenLabs Scribe v2 13.7%, 47 of 344 (95% interval 10.1 to 17.4): ours better, Detail Keeper minus ElevenLabs Scribe v2 −8.7 points (−12.5 to −5.1); Deepgram Nova-3 6.1%, 21 of 344 (95% interval 3.7 to 8.7): no measurable difference, Detail Keeper minus Deepgram Nova-3 −1.2 points (−4.1 to +1.8).
Taught by another person, untaught overwritten: Detail Keeper 0.6%, 2 of 348 (95% interval 0.0 to 1.5); ElevenLabs Scribe v2 1.7%, 6 of 348 (95% interval 0.6 to 3.2): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −1.1 points (−2.8 to +0.3); Deepgram Nova-3 1.7%, 6 of 348 (95% interval 0.6 to 3.2): no measurable difference, Detail Keeper minus Deepgram Nova-3 −1.1 points (−2.8 to +0.3).
Name-free insertions
lower is better · 0–10% · count
Parliament speech, name-free insertions: Detail Keeper 0.0%, 0 of 200 (95% interval 0.0 to 0.0); ElevenLabs Scribe v2 0.0%, 0 of 200 (95% interval 0.0 to 0.0): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 0.0 points (0.0 to 0.0); Deepgram Nova-3 0.0%, 0 of 200 (95% interval 0.0 to 0.0): no measurable difference, Detail Keeper minus Deepgram Nova-3 0.0 points (0.0 to 0.0).
Phone line, name-free insertions: Detail Keeper 0.0%, 0 of 200 (95% interval 0.0 to 0.0); ElevenLabs Scribe v2 0.5%, 1 of 200 (95% interval 0.0 to 1.6): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −0.5 points (−1.6 to 0.0); Deepgram Nova-3 0.0%, 0 of 200 (95% interval 0.0 to 0.0): no measurable difference, Detail Keeper minus Deepgram Nova-3 0.0 points (0.0 to 0.0).
Far from the microphone, name-free insertions: Detail Keeper 0.0%, 0 of 150 (95% interval 0.0 to 0.0); ElevenLabs Scribe v2 1.3%, 2 of 150 (95% interval 0.0 to 3.4): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −1.3 points (−3.4 to 0.0); Deepgram Nova-3 1.3%, 2 of 150 (95% interval 0.0 to 3.4): no measurable difference, Detail Keeper minus Deepgram Nova-3 −1.3 points (−3.4 to 0.0).
Street noise, name-free insertions: Detail Keeper 0.0%, 0 of 150 (95% interval 0.0 to 0.0); ElevenLabs Scribe v2 0.0%, 0 of 150 (95% interval 0.0 to 0.0): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 0.0 points (0.0 to 0.0); Deepgram Nova-3 0.0%, 0 of 150 (95% interval 0.0 to 0.0): no measurable difference, Detail Keeper minus Deepgram Nova-3 0.0 points (0.0 to 0.0).
Very quiet, taught up close, name-free insertions: Detail Keeper 1.7%, 2 of 120 (95% interval 0.0 to 4.2); ElevenLabs Scribe v2 1.7%, 2 of 120 (95% interval 0.0 to 4.2): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 0.0 points (−3.4 to +3.3); Deepgram Nova-3 0.0%, 0 of 120 (95% interval 0.0 to 0.0): no measurable difference, Detail Keeper minus Deepgram Nova-3 +1.7 points (0.0 to +4.2).
Short snippets, name-free insertions: Detail Keeper 1.1%, 4 of 376 (95% interval 0.2 to 2.2); ElevenLabs Scribe v2 6.4%, 24 of 376 (95% interval 3.8 to 9.2): ours better, Detail Keeper minus ElevenLabs Scribe v2 −5.3 points (−8.3 to −2.6); Deepgram Nova-3 0.3%, 1 of 376 (95% interval 0.0 to 0.9): no measurable difference, Detail Keeper minus Deepgram Nova-3 +0.8 points (0.0 to +1.8).
Ordinary-word error
lower is better · 0–80%
Parliament speech, ordinary-word error: Detail Keeper 6.3%, 389 of 6,130 (95% interval 5.6 to 7.1); ElevenLabs Scribe v2 7.0%, 429 of 6,130 (95% interval 6.0 to 8.1): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 −0.7 points (−1.7 to +0.1); Deepgram Nova-3 7.9%, 482 of 6,130 (95% interval 7.0 to 8.8): ours better, Detail Keeper minus Deepgram Nova-3 −1.5 points (−2.2 to −0.8).
Phone line, ordinary-word error: Detail Keeper 5.0%, 413 of 8,333 (95% interval 4.2 to 5.7); ElevenLabs Scribe v2 4.7%, 392 of 8,333 (95% interval 3.9 to 5.6): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 +0.3 points (−0.3 to +0.8); Deepgram Nova-3 7.1%, 588 of 8,333 (95% interval 6.2 to 8.1): ours better, Detail Keeper minus Deepgram Nova-3 −2.1 points (−2.9 to −1.4).
Far from the microphone, ordinary-word error: Detail Keeper 20.2%, 1,752 of 8,665 (95% interval 17.6 to 22.9); ElevenLabs Scribe v2 19.1%, 1,657 of 8,665 (95% interval 13.6 to 28.5): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 +1.1 points (−7.9 to +6.1); Deepgram Nova-3 32.4%, 2,804 of 8,665 (95% interval 28.8 to 36.0): ours better, Detail Keeper minus Deepgram Nova-3 −12.1 points (−14.8 to −9.7).
Street noise, ordinary-word error: Detail Keeper 7.2%, 621 of 8,634 (95% interval 6.4 to 8.1); ElevenLabs Scribe v2 6.1%, 527 of 8,634 (95% interval 5.2 to 7.1): ours worse, Detail Keeper minus ElevenLabs Scribe v2 +1.1 points (+0.3 to +1.9); Deepgram Nova-3 10.4%, 899 of 8,634 (95% interval 9.1 to 11.8): ours better, Detail Keeper minus Deepgram Nova-3 −3.2 points (−4.3 to −2.2).
Very quiet, taught up close, ordinary-word error: Detail Keeper 18.8%, 1,421 of 7,560 (95% interval 16.4 to 21.4); ElevenLabs Scribe v2 14.7%, 1,112 of 7,560 (95% interval 12.8 to 16.8): ours worse, Detail Keeper minus ElevenLabs Scribe v2 +4.1 points (+2.7 to +5.5); Deepgram Nova-3 37.6%, 2,841 of 7,560 (95% interval 33.4 to 41.7): ours better, Detail Keeper minus Deepgram Nova-3 −18.8 points (−22.2 to −15.4).
A sound-alike on the list, ordinary-word error: Detail Keeper 7.4%, 388 of 5,238 (95% interval 5.9 to 9.2); ElevenLabs Scribe v2 6.0%, 315 of 5,238 (95% interval 4.6 to 7.7): ours worse, Detail Keeper minus ElevenLabs Scribe v2 +1.4 points (+0.5 to +2.3); Deepgram Nova-3 11.1%, 581 of 5,238 (95% interval 8.6 to 14.0): ours better, Detail Keeper minus Deepgram Nova-3 −3.7 points (−5.6 to −2.0).
Short snippets, ordinary-word error: Detail Keeper 41.1%, 536 of 1,304 (95% interval 36.5 to 45.9); ElevenLabs Scribe v2 52.9%, 690 of 1,304 (95% interval 48.3 to 57.7): ours better, Detail Keeper minus ElevenLabs Scribe v2 −11.8 points (−15.7 to −7.9); Deepgram Nova-3 58.0%, 756 of 1,304 (95% interval 52.6 to 63.7): ours better, Detail Keeper minus Deepgram Nova-3 −16.9 points (−21.5 to −12.4).
Taught by another person, ordinary-word error: Detail Keeper 5.0%, 326 of 6,473 (95% interval 4.3 to 5.9); ElevenLabs Scribe v2 4.4%, 285 of 6,473 (95% interval 3.4 to 5.8): no measurable difference, Detail Keeper minus ElevenLabs Scribe v2 +0.6 points (−0.5 to +1.5); Deepgram Nova-3 6.6%, 426 of 6,473 (95% interval 5.6 to 7.6): ours better, Detail Keeper minus Deepgram Nova-3 −1.5 points (−2.2 to −0.9).
Method and caveats
A verdict compares ours with one service on the same items: "ours better" or "ours worse" only where the 95% interval of the paired difference excludes zero, "no measurable difference" otherwise. own-names-bench v1.3.0 (its data at Hugging Face tag v1.3.0; the results published at tag v1.3.1); our model v7c (v0.7.0), in Detail Keeper 0.1.2 and later. Runs: Detail Keeper ; ElevenLabs Scribe v2 and ; Deepgram Nova-3 .
Sealed audiobooks
20 LibriVox audiobooks, each with its own reader, none of them used before by the benchmark or in our training: 1,100 mentions (777 of invented names, 323 of traditional ones) and 1,200 name-free clips (193 minutes). Each clip runs with its book's own list of 9 to 176 names. Each name is taught from the same reader: their best-aligned isolated reading of it elsewhere in the book, never a clip it is tested on.
Sealed conversations
Eight conditions, built from the test splits of VoxPopuli (European Parliament speech) and LibriTTS-R (read speech), with new noise from FSD50K and simulated rooms: 2,327 mentions and 1,196 name-free pieces (102.9 minutes), 40 names per list. Each name is taught from one recording of it: in all but one condition, the same speaker for 1,359 of 1,979 mentions and another person for the rest; in the condition taught by another person, always another person.
- Parliament speech
- Spontaneous European Parliament speech (VoxPopuli), clean, as recorded: 132 mentions; 200 name-free pieces (25.7 minutes).
- Phone line
- Each clip and every taught recording through a phone line (300 to 3,400 Hz, 8 kHz, G.711, GSM or Opus, lost packets): 270 mentions; 200 name-free pieces (22.0 minutes).
- Far from the microphone
- A simulated room, 1 to 5 m from the microphone, with noise, babble or both, and left quiet: 333 mentions; 150 name-free pieces (16.8 minutes).
- Street noise
- Traffic, vehicles, sirens, crowds, rain, wind and the subway at -2 to 15 dB below or above the speech: 320 mentions; 150 name-free pieces (18.7 minutes).
- Very quiet, taught up close
- A far-off clip at -50 to -36 dBFS, taught from loud close-talk recordings: 300 mentions; 120 name-free pieces (14.0 minutes).
- A sound-alike on the list
- A real name one or two sounds from the spoken one is always on the list: 280 mentions; no name-free pieces.
- Short snippets
- 0.5 to 2 seconds around the name, and name-free pieces of 0.4 to 2 seconds: 344 mentions; 376 name-free pieces (5.7 minutes).
- Taught by another person
- Every name taught from another person's recording, clean or with indoor or street noise: 348 mentions; no name-free pieces.
Read with care
Sealed audiobooks
- Sealed on 27 Oct 2026, after v7c had been chosen and frozen, then scored once. It measures the frozen system; it did not help choose it.
- 11 of the 20 readers are LibriSpeech speakers, and 5 of the books' readings (by the suite's own reader) are in LibriSpeech. None of that is in our adapter's training data, but the base model, Qwen3-ASR, and the hosted services may have trained on it.
- With no list at all, the names are written right by Detail Keeper 22.0%, ElevenLabs Scribe v2 29.8% and Deepgram Nova-3 16.9%.
Sealed conversations
- Sealed on 16 Oct 2026, before any of our final adapters existed, then scored once per final system.
- ElevenLabs' run is its run of 16 Oct 2026 on the same sealed data, reused rather than run again.
- There is no pooled interval, so whether a difference is real is shown per condition only. Totals are plain counts.
- VoxPopuli is held out by speaker, not by debate: 265 of the 404 parliament utterances come from debates in which other speakers are in our training data.
The systems, as configured
Hosted services change: each result is the service on the date of its run. Our adapter is trained on public-domain, CC0 and CC BY speech and noise, and on synthetic speech in adult LibriTTS readers' voices, on top of Alibaba's Qwen3-ASR-1.7B; the name spotter uses Meta's wav2vec2 and Microsoft's WavLM (CC BY-SA 3.0, downloaded from Microsoft). The runtime in the app: virgil-coach/detailkeeper-runtime.
Detail Keeper
- Model
- virgil-coach/own-names-qwen3-asr v0.7.0-rc-v7c (v7c) on Qwen3-ASR-1.7B, with the spotter, guard and re-read; the Detail Keeper engine (MLX, 8-bit) on Apple silicon
- Given
- The spelling, and one recording of each name (taught as in the app).
- How much of the list
- no limit: the spotter ranks every name
- Runs
- and
ElevenLabs Scribe v2
- Model
- scribe_v2
- Given
- The spelling only, as keyterms.
- How much of the list
- drop terms that break a per-term rule, then keep the first 1000 (most frequent in the book first)
- Runs
- , and
Deepgram Nova-3
- Model
- nova-3
- Given
- The spelling only, as keyterm prompting.
- How much of the list
- the longest prefix of the list (most frequent first) the service accepts, found by binary search with half a second of silence (refused requests are not billed)
- Runs
- and
Every number
Every set, including the v1.3.0 suites and the development set we chose our model on. Percent with 95% intervals (a bootstrap over books, calls, recordings or list groups), the counts behind each rate (for word error, errors of words), and each service's paired difference from ours. The best rate in a row is bold.
| Task and metric | Detail Keeper | ElevenLabs Scribe v2 | Deepgram Nova-3 |
|---|---|---|---|
| Sealed audiobooksSealed, scored once | |||
| Names righthigher is better · 1,100 mentions | 89.6 [83.4, 94.4] 986 of 1,100 | 88.2 [79.4, 95.1] 970 of 1,100no measurable difference, +1.5 pts [−1.9, +4.7] | 78.6 [65.1, 88.9] 865 of 1,100ours better, +11.0 pts [+3.8, +19.1] |
| Untaught overwrittenlower is better · 1,100 mentions | 3.6 [1.1, 6.6] 40 of 1,100 | 1.5 [0.5, 2.4] 16 of 1,100ours worse, +2.2 pts [+0.1, +4.6] | 5.9 [3.9, 7.6] 65 of 1,100no measurable difference, −2.3 pts [−4.6, 0.0] |
| Name-free insertionslower is better · 1,200 name-free items | 0.2 [0.0, 0.4] 2 of 1,200 | 0.2 [0.0, 0.4] 2 of 1,200no measurable difference, 0.0 pts [−0.2, +0.2] | 1.3 [0.4, 2.7] 16 of 1,200ours better, −1.2 pts [−2.3, −0.3] |
| Ordinary-word errorlower is better · 55,743 words | 2.1 [1.7, 2.6] 1,178 of 55,743 | 1.5 [1.2, 1.9] 854 of 55,743ours worse, +0.6 pts [+0.4, +0.7] | 3.1 [2.4, 3.8] 1,703 of 55,743ours better, −0.9 pts [−1.4, −0.5] |
| Results fileand run dates | results/books13/scores/summary.json (books13-v7c-mlx) | results/books13/scores/summary.json (books13-elevenlabs) | results/books13/scores/summary.json (books13-deepgram) |
| Sealed conversationsSealed, scored once · one section per condition | |||
| Parliament speech | |||
| Names righthigher is better · 132 mentions | 98.5 [96.2, 100.0] 130 of 132 | 97.7 [94.8, 100.0] 129 of 132no measurable difference, +0.8 pts [−2.3, +4.2] | 96.2 [92.5, 99.2] 127 of 132no measurable difference, +2.3 pts [−1.5, +6.5] |
| Untaught overwrittenlower is better · 132 mentions | 0.0 [0.0, 0.0] 0 of 132 | 0.8 [0.0, 2.4] 1 of 132no measurable difference, −0.8 pts [−2.4, 0.0] | 2.3 [0.0, 5.1] 3 of 132no measurable difference, −2.3 pts [−5.1, 0.0] |
| Name-free insertionslower is better · 200 name-free items | 0.0 [0.0, 0.0] 0 of 200 | 0.0 [0.0, 0.0] 0 of 200no measurable difference, 0.0 pts [0.0, 0.0] | 0.0 [0.0, 0.0] 0 of 200no measurable difference, 0.0 pts [0.0, 0.0] |
| Ordinary-word errorlower is better · 6,130 words | 6.3 [5.6, 7.1] 389 of 6,130 | 7.0 [6.0, 8.1] 429 of 6,130no measurable difference, −0.7 pts [−1.7, +0.1] | 7.9 [7.0, 8.8] 482 of 6,130ours better, −1.5 pts [−2.2, −0.8] |
| Phone line | |||
| Names righthigher is better · 270 mentions | 98.1 [96.3, 99.6] 265 of 270 | 97.0 [94.9, 98.9] 262 of 270no measurable difference, +1.1 pts [−1.1, +3.4] | 90.4 [86.7, 93.6] 244 of 270ours better, +7.8 pts [+4.5, +11.4] |
| Untaught overwrittenlower is better · 270 mentions | 2.6 [0.8, 4.7] 7 of 270 | 8.1 [5.1, 11.4] 22 of 270ours better, −5.6 pts [−8.8, −2.4] | 9.3 [5.9, 12.8] 25 of 270ours better, −6.7 pts [−10.1, −3.3] |
| Name-free insertionslower is better · 200 name-free items | 0.0 [0.0, 0.0] 0 of 200 | 0.5 [0.0, 1.6] 1 of 200no measurable difference, −0.5 pts [−1.6, 0.0] | 0.0 [0.0, 0.0] 0 of 200no measurable difference, 0.0 pts [0.0, 0.0] |
| Ordinary-word errorlower is better · 8,333 words | 5.0 [4.2, 5.7] 413 of 8,333 | 4.7 [3.9, 5.6] 392 of 8,333no measurable difference, +0.3 pts [−0.3, +0.8] | 7.1 [6.2, 8.1] 588 of 8,333ours better, −2.1 pts [−2.9, −1.4] |
| Far from the microphone | |||
| Names righthigher is better · 333 mentions | 82.9 [78.3, 87.2] 276 of 333 | 88.0 [84.4, 91.5] 293 of 333ours worse, −5.1 pts [−9.1, −1.4] | 60.7 [54.7, 66.7] 202 of 333ours better, +22.2 pts [+17.2, +27.3] |
| Untaught overwrittenlower is better · 333 mentions | 5.4 [3.1, 7.9] 18 of 333 | 11.4 [8.3, 14.8] 38 of 333ours better, −6.0 pts [−9.4, −2.7] | 9.9 [6.9, 13.3] 33 of 333ours better, −4.5 pts [−8.2, −0.9] |
| Name-free insertionslower is better · 150 name-free items | 0.0 [0.0, 0.0] 0 of 150 | 1.3 [0.0, 3.4] 2 of 150no measurable difference, −1.3 pts [−3.4, 0.0] | 1.3 [0.0, 3.4] 2 of 150no measurable difference, −1.3 pts [−3.4, 0.0] |
| Ordinary-word errorlower is better · 8,665 words | 20.2 [17.6, 22.9] 1,752 of 8,665 | 19.1 [13.6, 28.5] 1,657 of 8,665no measurable difference, +1.1 pts [−7.9, +6.1] | 32.4 [28.8, 36.0] 2,804 of 8,665ours better, −12.1 pts [−14.8, −9.7] |
| Street noise | |||
| Names righthigher is better · 320 mentions | 93.8 [90.9, 96.4] 300 of 320 | 95.6 [93.3, 97.7] 306 of 320no measurable difference, −1.9 pts [−4.9, +1.2] | 85.3 [81.4, 89.2] 273 of 320ours better, +8.4 pts [+5.0, +12.0] |
| Untaught overwrittenlower is better · 320 mentions | 5.6 [3.3, 8.3] 18 of 320 | 6.9 [4.3, 9.6] 22 of 320no measurable difference, −1.2 pts [−4.0, +1.6] | 10.6 [7.4, 13.9] 34 of 320ours better, −5.0 pts [−8.3, −1.8] |
| Name-free insertionslower is better · 150 name-free items | 0.0 [0.0, 0.0] 0 of 150 | 0.0 [0.0, 0.0] 0 of 150no measurable difference, 0.0 pts [0.0, 0.0] | 0.0 [0.0, 0.0] 0 of 150no measurable difference, 0.0 pts [0.0, 0.0] |
| Ordinary-word errorlower is better · 8,634 words | 7.2 [6.4, 8.1] 621 of 8,634 | 6.1 [5.2, 7.1] 527 of 8,634ours worse, +1.1 pts [+0.3, +1.9] | 10.4 [9.1, 11.8] 899 of 8,634ours better, −3.2 pts [−4.3, −2.2] |
| Very quiet, taught up close | |||
| Names righthigher is better · 300 mentions | 78.7 [73.7, 83.5] 236 of 300 | 84.3 [79.8, 88.7] 253 of 300ours worse, −5.7 pts [−9.9, −1.4] | 54.0 [48.0, 59.9] 162 of 300ours better, +24.7 pts [+19.1, +30.2] |
| Untaught overwrittenlower is better · 300 mentions | 5.3 [3.0, 7.9] 16 of 300 | 11.0 [7.8, 14.4] 33 of 300ours better, −5.7 pts [−8.9, −2.5] | 8.3 [5.3, 11.7] 25 of 300no measurable difference, −3.0 pts [−6.7, +0.6] |
| Name-free insertionslower is better · 120 name-free items | 1.7 [0.0, 4.2] 2 of 120 | 1.7 [0.0, 4.2] 2 of 120no measurable difference, 0.0 pts [−3.4, +3.3] | 0.0 [0.0, 0.0] 0 of 120no measurable difference, +1.7 pts [0.0, +4.2] |
| Ordinary-word errorlower is better · 7,560 words | 18.8 [16.4, 21.4] 1,421 of 7,560 | 14.7 [12.8, 16.8] 1,112 of 7,560ours worse, +4.1 pts [+2.7, +5.5] | 37.6 [33.4, 41.7] 2,841 of 7,560ours better, −18.8 pts [−22.2, −15.4] |
| A sound-alike on the list | |||
| Names righthigher is better · 280 mentions | 92.1 [88.9, 95.1] 258 of 280 | 92.5 [89.3, 95.4] 259 of 280no measurable difference, −0.4 pts [−3.9, +3.2] | 85.4 [80.7, 89.7] 239 of 280ours better, +6.8 pts [+2.6, +11.2] |
| Untaught overwrittenlower is better · 280 mentions | 5.4 [2.8, 8.2] 15 of 280 | 12.5 [8.6, 16.9] 35 of 280ours better, −7.1 pts [−11.9, −2.8] | 15.4 [11.4, 19.5] 43 of 280ours better, −10.0 pts [−14.0, −6.1] |
| Ordinary-word errorlower is better · 5,238 words | 7.4 [5.9, 9.2] 388 of 5,238 | 6.0 [4.6, 7.7] 315 of 5,238ours worse, +1.4 pts [+0.5, +2.3] | 11.1 [8.6, 14.0] 581 of 5,238ours better, −3.7 pts [−5.6, −2.0] |
| Short snippets | |||
| Names righthigher is better · 344 mentions | 85.2 [81.1, 89.0] 293 of 344 | 84.6 [80.2, 88.8] 291 of 344no measurable difference, +0.6 pts [−4.2, +5.4] | 62.2 [56.5, 67.6] 214 of 344ours better, +23.0 pts [+17.7, +28.5] |
| Untaught overwrittenlower is better · 344 mentions | 4.9 [2.8, 7.3] 17 of 344 | 13.7 [10.1, 17.4] 47 of 344ours better, −8.7 pts [−12.5, −5.1] | 6.1 [3.7, 8.7] 21 of 344no measurable difference, −1.2 pts [−4.1, +1.8] |
| Name-free insertionslower is better · 376 name-free items | 1.1 [0.2, 2.2] 4 of 376 | 6.4 [3.8, 9.2] 24 of 376ours better, −5.3 pts [−8.3, −2.6] | 0.3 [0.0, 0.9] 1 of 376no measurable difference, +0.8 pts [0.0, +1.8] |
| Ordinary-word errorlower is better · 1,304 words | 41.1 [36.5, 45.9] 536 of 1,304 | 52.9 [48.3, 57.7] 690 of 1,304ours better, −11.8 pts [−15.7, −7.9] | 58.0 [52.6, 63.7] 756 of 1,304ours better, −16.9 pts [−21.5, −12.4] |
| Taught by another person | |||
| Names righthigher is better · 348 mentions | 99.1 [98.0, 100.0] 345 of 348 | 99.4 [98.5, 100.0] 346 of 348no measurable difference, −0.3 pts [−1.5, +0.9] | 97.7 [96.1, 99.1] 340 of 348no measurable difference, +1.4 pts [−0.3, +3.3] |
| Untaught overwrittenlower is better · 348 mentions | 0.6 [0.0, 1.5] 2 of 348 | 1.7 [0.6, 3.2] 6 of 348no measurable difference, −1.1 pts [−2.8, +0.3] | 1.7 [0.6, 3.2] 6 of 348no measurable difference, −1.1 pts [−2.8, +0.3] |
| Ordinary-word errorlower is better · 6,473 words | 5.0 [4.3, 5.9] 326 of 6,473 | 4.4 [3.4, 5.8] 285 of 6,473no measurable difference, +0.6 pts [−0.5, +1.5] | 6.6 [5.6, 7.6] 426 of 6,473ours better, −1.5 pts [−2.2, −0.9] |
| Results fileand run dates | results/conv13/scores/summary.json (conv13-v7c-mlx) | results/conv13/scores/summary.json (conv13-elevenlabs-2026-10-16) | results/conv13/scores/summary.json (conv13-deepgram) |
| Callsin own-names-bench since v1.2.0 | |||
| Names righthigher is better · 706 mentions | 95.3 [93.4, 97.0] 673 of 706 | 95.3 [93.3, 97.0] 673 of 706no measurable difference, 0.0 pts [−2.2, +2.2] | 91.9 [89.5, 94.2] 649 of 706ours better, +3.4 pts [+0.9, +5.9] |
| Untaught overwrittenlower is better · 706 mentions | 1.0 [0.2, 2.1] 7 of 706 | 1.6 [0.5, 3.0] 11 of 706no measurable difference, −0.6 pts [−1.6, +0.3] | 1.7 [0.7, 3.0] 12 of 706no measurable difference, −0.7 pts [−2.0, +0.4] |
| Name-free insertionslower is better · 417 name-free items | 1.7 [0.5, 2.9] 7 of 417 | 1.4 [0.5, 2.6] 6 of 417no measurable difference, +0.2 pts [−0.5, +1.0] | 1.2 [0.2, 2.4] 5 of 417no measurable difference, +0.5 pts [−0.5, +1.4] |
| Ordinary-word errorlower is better · 64,152 words | 9.8 [9.1, 10.6] 6,295 of 64,152 | 9.1 [8.4, 9.9] 5,830 of 64,152ours worse, +0.7 pts [+0.4, +1.1] | 10.2 [9.4, 11.0] 6,517 of 64,152no measurable difference, −0.3 pts [−0.7, 0.0] |
| Results fileand run dates | results/scores/ours-mlx-v7c.json | results/scores/elevenlabs-2026-10-26.json | results/scores/deepgram-2026-10-26.json |
| Voice questions, another speakerin own-names-bench since v1.2.0 · Taught by another person's recording | |||
| Names righthigher is better · 1,420 mentions | 93.8 [92.0, 95.5] 1,332 of 1,420 | 98.5 [97.6, 99.2] 1,399 of 1,420ours worse, −4.7 pts [−6.5, −3.1] | 89.4 [87.2, 91.5] 1,270 of 1,420ours better, +4.4 pts [+2.2, +6.5] |
| Untaught overwrittenlower is better · 1,420 mentions | 0.1 [0.0, 0.4] 2 of 1,420 | 0.6 [0.1, 1.3] 8 of 1,420no measurable difference, −0.4 pts [−1.0, 0.0] | 1.5 [0.8, 2.3] 21 of 1,420ours better, −1.3 pts [−2.1, −0.7] |
| Name-free insertionslower is better · 320 name-free items | 0.0 [0.0, 0.0] 0 of 320 | 0.9 [0.0, 2.2] 3 of 320no measurable difference, −0.9 pts [−2.2, 0.0] | 0.0 [0.0, 0.0] 0 of 320no measurable difference, 0.0 pts [0.0, 0.0] |
| Ordinary-word errorlower is better · 9,824 words | 8.9 [7.7, 10.3] 879 of 9,824 | 13.8 [11.7, 16.2] 1,359 of 9,824ours better, −4.9 pts [−7.0, −3.0] | 17.3 [15.5, 19.3] 1,702 of 9,824ours better, −8.4 pts [−10.0, −6.9] |
| Results fileand run dates | results/scores/ours-mlx-v7c.json | results/scores/elevenlabs-2026-10-26.json | results/scores/deepgram-2026-10-26.json |
| Far-field meetings, name-freein own-names-bench since v1.2.0 | |||
| Name-free insertionslower is better · 119 name-free items | 0.0 [0.0, 0.0] 0 of 119 | 0.0 [0.0, 0.0] 0 of 119no measurable difference, 0.0 pts [0.0, 0.0] | 0.0 [0.0, 0.0] 0 of 119no measurable difference, 0.0 pts [0.0, 0.0] |
| Ordinary-word errorlower is better · 10,657 words | 19.0 [16.4, 20.9] 2,026 of 10,657 | 16.7 [15.0, 18.2] 1,782 of 10,657ours worse, +2.3 pts [+0.7, +3.9] | 19.0 [17.5, 20.9] 2,029 of 10,657no measurable difference, 0.0 pts [−2.2, +2.2] |
| Results fileand run dates | results/scores/ours-mlx-v7c.json | results/scores/elevenlabs-2026-10-26.json | results/scores/deepgram-2026-10-26.json |
| Public meetingsin own-names-bench since v1.2.0 | |||
| Names righthigher is better · 714 mentions | 91.6 [88.3, 94.2] 654 of 714 | 93.1 [89.3, 96.2] 665 of 714no measurable difference, −1.5 pts [−2.8, 0.0] | 84.9 [79.1, 89.6] 606 of 714ours better, +6.7 pts [+4.1, +9.9] |
| Untaught overwrittenlower is better · 714 mentions | 0.6 [0.1, 1.3] 4 of 714 | 0.3 [0.0, 0.8] 2 of 714no measurable difference, +0.3 pts [0.0, +0.8] | 1.5 [0.5, 3.3] 11 of 714no measurable difference, −1.0 pts [−2.3, 0.0] |
| Ordinary-word errorlower is better · 14,227 words | 11.9 [9.6, 15.1] 1,688 of 14,227 | 11.6 [9.6, 14.5] 1,654 of 14,227no measurable difference, +0.2 pts [−0.3, +1.0] | 14.0 [11.5, 17.5] 1,995 of 14,227ours better, −2.2 pts [−2.9, −1.4] |
| Results fileand run dates | results/scores/ours-mlx-v7c.json | results/scores/elevenlabs-2026-10-26.json | results/scores/deepgram-2026-10-26.json |
| Development set · Audiobooks: clipsUsed to choose between our candidate models (12 logged looks), so our figure on it is optimistic. · in own-names-bench since v1.0.0 | |||
| Names righthigher is better · 1,107 mentions | 92.6 [88.8, 96.5] 1,025 of 1,107 | 92.8 [88.5, 97.2] 1,027 of 1,107no measurable difference, −0.2 pts [−2.7, +1.9] | 75.1 [61.8, 89.5] 831 of 1,107ours better, +17.5 pts [+5.9, +27.2] |
| Untaught overwrittenlower is better · 1,107 mentions | 4.6 [1.1, 8.0] 51 of 1,107 | 2.9 [1.2, 4.8] 32 of 1,107no measurable difference, +1.7 pts [−1.3, +4.4] | 8.5 [5.9, 11.1] 94 of 1,107ours better, −3.9 pts [−7.4, −1.3] |
| Name-free insertionslower is better · 408 name-free items | 1.2 [0.2, 2.5] 5 of 408 | 0.2 [0.0, 0.7] 1 of 408no measurable difference, +1.0 pts [−0.2, +2.2] | 1.7 [0.5, 2.9] 7 of 408no measurable difference, −0.5 pts [−2.0, +1.0] |
| Ordinary-word errorlower is better · 39,814 words | 2.8 [2.3, 3.3] 1,105 of 39,814 | 1.7 [1.4, 1.9] 665 of 39,814ours worse, +1.1 pts [+0.7, +1.5] | 4.2 [3.1, 5.1] 1,672 of 39,814ours better, −1.4 pts [−2.0, −0.7] |
| Results fileand run dates | results/scores/ours-mlx-v7c.json | results/scores/elevenlabs-2026-10-26.json | results/scores/deepgram-2026-10-26.json |
Reproduce: own-names-bench v1.3.1 publishes every pass, and its README rebuilds the sealed reports without API keys (data: Hugging Face v1.3.0; rules: CLAIMS.md; our looks: LOOKS.md).
Private, on the Mac: audio and text never leave it.
Our own model, open weights: virgil-coach/own-names-qwen3-asr v0.7.0, Apache 2.0.
English only.