Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR
For ASR researchers and practitioners, this work addresses the problem of adapting models to disfluent speech while preserving general-domain knowledge, but the results are incremental as they primarily analyze trade-offs rather than achieving new SOTA.
The paper tackles disfluent speech recognition by introducing explicit disfluency tokens into a pretrained ASR model and applying continual learning to adapt to varying disfluency distributions without catastrophic forgetting. It identifies a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism across CL methods.
Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.