SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR
Provides a more accurate evaluation method for ASR systems in agglutinative languages, addressing a known bottleneck in error analysis for Indic languages.
SCRIBE introduces a diagnostic framework for Indic ASR that decomposes errors into lexical, punctuation, numeral, and domain-entity rates using sandhi-tolerant alignment, outperforming WER in aligning with expert judgment. The framework includes benchmarks and open-weight models for three Indic languages.
Automatic speech recognition replaces typing only when correction costs less than manual entry, a threshold determined by error types, not counts: fixing a misrecognized domain term costs far more than inserting a comma. Word error rate (WER) fails on two fronts: it collapses distinct error categories into a single scalar, and it structurally penalizes agglutinative languages where valid sandhi merges inflate scores. We introduce SCRIBE, a diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi-tolerant alignment with domain vocabulary injection. Human validation confirms SCRIBE aligns with expert judgment where WER does not. We release SCRIBE, an LLM curation pipeline, benchmarks, and open-weight rich transcription models for Hindi, Malayalam, and Kannada.