Script Collapse in Multilingual ASR: Defining and Measuring Script Fidelity Rate
For researchers and practitioners of multilingual ASR, this work highlights a critical blind spot in evaluation and provides a simple metric to detect script collapse, which is a practical problem for deployment in diverse linguistic contexts.
The paper identifies a failure mode in multilingual ASR where models produce fluent output in the wrong writing system, which WER cannot detect. They propose Script Fidelity Rate (SFR) and measure script collapse across six languages and nine models, finding 34% of model-language pairs exhibit collapse (SFR < 10%), with MMS-1B and SeamlessM4T-v2 maintaining SFR > 99%.
Word error rate (WER) is the dominant metric for automatic speech recognition, yet it cannot detect a systematic failure mode: models that produce fluent output in the wrong writing system. We define Script Fidelity Rate (SFR), the fraction of hypothesis characters in the target script block, computable without reference transcriptions, and report the first systematic measurement of script collapse across six languages spanning four writing systems (Pashto, Urdu, Hindi, Bengali, Malayalam, Somali) and nine ASR models on FLEURS test sets. Across 53 evaluated model-language pairs, 18 (34%; 95% Wilson CI: 23-47%) exhibit script collapse (SFR < 10%); MMS-1B and SeamlessM4T-v2 maintain SFR above 99% on every language evaluated, confirming that SFR correctly identifies high fidelity where it is present. We identify three distinct collapse patterns: Latin phonetic substitution (smaller Whisper on Indic languages), Arabic substitution for Somali's Latin-script orthography, and Devanagari substitution where larger Whisper models treat all Indic audio as Hindi, a failure present even in Whisper large-v3.