Earnings-22: A Practical Benchmark for Accents in the Wild
This provides a practical benchmark for accents in the wild, addressing a gap for researchers and developers in ASR, though it is incremental as it focuses on data collection and analysis.
The authors tackled the lack of real-world accented speech corpora for benchmarking ASR systems by introducing Earnings-22, a 119-hour corpus of English earnings calls, and found that key speech features impact model performance more for certain accents, with variations across 4 commercial models.
Modern automatic speech recognition (ASR) systems have achieved superhuman Word Error Rate (WER) on many common corpora despite lacking adequate performance on speech in the wild. Beyond that, there is a lack of real-world, accented corpora to properly benchmark academic and commercial models. To ensure this type of speech is represented in ASR benchmarking, we present Earnings-22, a 125 file, 119 hour corpus of English-language earnings calls gathered from global companies. We run a comparison across 4 commercial models showing the variation in performance when taking country of origin into consideration. Looking at hypothesis transcriptions, we explore errors common to all ASR systems tested. By examining Individual Word Error Rate (IWER), we find that key speech features impact model performance more for certain accents than others. Earnings-22 provides a free-to-use benchmark of real-world, accented audio to bridge academic and industrial research.