"Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most
This addresses a critical gap in real-world reliability of speech systems for high-stakes tasks, particularly affecting linguistically diverse users, with incremental improvements through fine-tuning.
The study tackled the problem of speech recognition models failing on short, high-stakes utterances like U.S. street names, finding an average transcription error rate of 44%, and introduced a synthetic data approach that improved accuracy by nearly 60% for non-English primary speakers.
Despite speech recognition systems achieving low word error rates on standard benchmarks, they often fail on short, high-stakes utterances in real-world deployments. Here, we study this failure mode in a high-stakes task: the transcription of U.S. street names as spoken by U.S. participants. We evaluate 15 models from OpenAI, Deepgram, Google, and Microsoft on recordings from linguistically diverse U.S. speakers and find an average transcription error rate of 44%. We quantify the downstream impact of failed transcriptions by geographic locations and show that mis-transcriptions systematically cause errors for all speakers, but that routing distance errors are twice as large for non-English primary speakers compared to English primary speakers. To mitigate this harm, we introduce a synthetic data generation approach that produces diverse pronunciations of named entities using open-source text-to-speech models. Fine-tuning with less than 1,000 synthetic samples improves street name transcription accuracy by nearly 60% (relative to base models) for non-English primary speakers. Our results highlight a critical gap between benchmark performance and real-world reliability in speech systems and demonstrate a simple, scalable path to reducing high-stakes transcription errors.