Data-Efficient ASR Personalization for Non-Normative Speech Using an Uncertainty-Based Phoneme Difficulty Score for Guided Sampling
This addresses the problem of degraded ASR performance for individuals with speech impairments caused by conditions like cerebral palsy, offering a practical framework for personalized and inclusive ASR.
This work tackles the problem of ASR systems struggling with non-normative speech from individuals with impairments by introducing a data-efficient personalization method that uses phoneme-level uncertainty to guide fine-tuning. The results show that this clinically-validated, uncertainty-guided sampling significantly improves ASR accuracy, with the model-derived uncertainty strongly correlating with expert clinical assessments of speech difficulty.
Automatic speech recognition (ASR) systems struggle with non-normative speech from individuals with impairments caused by conditions like cerebral palsy or structural anomalies. The high acoustic variability and scarcity of training data severely degrade model performance. This work introduces a data-efficient personalization method that quantifies phoneme-level uncertainty to guide fine-tuning. We leverage Monte Carlo Dropout to estimate which phonemes a model finds most difficult and use these estimates for a targeted oversampling strategy. We validate our method on English and German datasets. Crucially, we demonstrate that our model-derived uncertainty strongly correlates with phonemes identified as challenging in an expert clinical logopedic report, marking, to our knowledge, the first work to successfully align model uncertainty with expert assessment of speech difficulty. Our results show that this clinically-validated, uncertainty-guided sampling significantly improves ASR accuracy, delivering a practical framework for personalized and inclusive ASR.