Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
This work improves evaluation metrics for under-resourced Indian languages, though it is incremental as it builds on existing methods like WER-SN.
The paper tackles the challenge of evaluating speech recognition systems for Indian languages by addressing spelling variations and non-standard spellings, proposing a framework that reduces pessimistic error rates by an average of 6.3 points and aligns more closely with human perception.
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.