Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levels
This work addresses fairness and efficacy issues in ASR for diverse languages, though it is incremental as it focuses on analysis rather than new methods.
The paper analyzed Whisper's multilingual decoder by examining sub-token hypotheses across languages with different resource levels, revealing that higher-resource languages have higher correct token likelihood and confidence, while lower-resource languages show distinct clustering patterns in sub-token usage.
While large multilingual automatic speech recognition (ASR) models achieve remarkable performance, the internal mechanisms of the end-to-end pipeline, particularly concerning fairness and efficacy across languages, remain underexplored. This paper introduces a fine-grained analysis of Whisper's multilingual decoder, examining its sub-token hypotheses during transcription across languages with various resource levels. Our method traces the beam search path, capturing sub-token guesses and their associated probabilities. Results reveal that higher resource languages benefit from higher likelihood of the correct token being top-ranked, greater confidence, lower predictive entropy, and more diverse alternative candidates. Lower resource languages fare worse on these metrics, but also exhibit distinct clustering patterns in sub-token usage sometimes influenced by typology in our PCA and t-SNE analysis. This sub-token probing uncovers systematic decoding disparities masked by aggregate error rates and points towards targeted interventions to ameliorate the imbalanced development of speech technology.