Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation
This addresses the problem of inconsistent and limited ASR evaluation for researchers and practitioners, though it is incremental as it builds on existing benchmarks and methods.
The authors tackled the lack of reproducible and transparent evaluation in ASR by introducing the Open ASR Leaderboard, which benchmarks over 60 systems across 11 datasets, showing that Conformer encoders with LLM decoders achieve the best average WER for English but are slower, while CTC and TDT decoders offer better efficiency.
Despite rapid progress, ASR evaluation remains saturated with short-form English, and efficiency is rarely reported. We present the Open ASR Leaderboard, a fully reproducible benchmark and interactive leaderboard comparing 60+ open-source and proprietary systems across 11 datasets, including dedicated multilingual and long-form tracks. We standardize text normalization and report both word error rate (WER) and inverse real-time factor (RTFx), enabling fair accuracy-efficiency comparisons. For English transcription, Conformer encoders paired with LLM decoders achieve the best average WER but are slower, while CTC and TDT decoders deliver much better RTFx, making them attractive for long-form and offline use. Whisper-derived encoders fine-tuned for English improve accuracy but often trade off multilingual coverage. All code and dataset loaders are open-sourced to support transparent, extensible evaluation.