Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
This addresses the challenge of adapting speech recognition models to low-resource domains, though it is an incremental improvement over existing continued pre-training methods.
The paper tackles the problem of overfitting in continued self-supervised pre-training for low-resource automatic speech recognition by proposing Stable Distillation, which uses self-distillation as regularization, resulting in performance gains of 0.8 to 7 WER over baselines.
Continued self-supervised (SSL) pre-training for adapting existing SSL models to the target domain has shown to be extremely effective for low-resource Automatic Speech Recognition (ASR). This paper proposes Stable Distillation, a simple and novel approach for SSL-based continued pre-training that boosts ASR performance in the target domain where both labeled and unlabeled data are limited. Stable Distillation employs self-distillation as regularization for continued pre-training, alleviating the over-fitting issue, a common problem continued pre-training faces when the source and target domains differ. Specifically, first, we perform vanilla continued pre-training on an initial SSL pre-trained model on the target domain ASR dataset and call it the teacher. Next, we take the same initial pre-trained model as a student to perform continued pre-training while enforcing its hidden representations to be close to that of the teacher (via MSE loss). This student is then used for downstream ASR fine-tuning on the target dataset. In practice, Stable Distillation outperforms all our baselines by 0.8 - 7 WER when evaluated in various experimental settings.