The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
This addresses the problem of inclusive ASR benchmarking for diverse language varieties, though it is incremental as it builds on existing challenge frameworks.
The paper tackles the unequal distribution of multilingual ASR improvements across languages and varieties by introducing the ML-SUPERB 2.0 Challenge, which constructed a test suite with 200+ languages/accents/dialects and received submissions that outperformed baselines, with the best achieving up to 30.2% lower CER and 23% higher LID accuracy.
Recent improvements in multilingual ASR have not been equally distributed across languages and language varieties. To advance state-of-the-art (SOTA) ASR models, we present the Interspeech 2025 ML-SUPERB 2.0 Challenge. We construct a new test suite that consists of data from 200+ languages, accents, and dialects to evaluate SOTA multilingual speech models. The challenge also introduces an online evaluation server based on DynaBench, allowing for flexibility in model design and architecture for participants. The challenge received 5 submissions from 3 teams, all of which outperformed our baselines. The best-performing submission achieved an absolute improvement in LID accuracy of 23% and a reduction in CER of 18% when compared to the best baseline on a general multilingual test set. On accented and dialectal data, the best submission obtained 30.2% lower CER and 15.7% higher LID accuracy, showing the importance of community challenges in making speech technologies more inclusive.