SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training
This work addresses the need for efficient and noise-robust speech pre-training methods for real-world applications, representing an incremental improvement over existing approaches.
The paper tackles the problem of speech pre-training by introducing SPIRAL, a self-supervised method that learns denoising representations through a teacher-student framework with perturbation and contrastive loss, achieving competitive or better results than wav2vec 2.0 with 80% and 65% reductions in training cost for BASE and LARGE models, respectively, and improving noise robustness with 9.0% to 13.3% relative word error rate reductions on noisy test data.
We introduce a new approach for speech pre-training named SPIRAL which works by learning denoising representation of perturbed data in a teacher-student framework. Specifically, given a speech utterance, we first feed the utterance to a teacher network to obtain corresponding representation. Then the same utterance is perturbed and fed to a student network. The student network is trained to output representation resembling that of the teacher. At the same time, the teacher network is updated as moving average of student's weights over training steps. In order to prevent representation collapse, we apply an in-utterance contrastive loss as pre-training objective and impose position randomization on the input to the teacher. SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training method wav2vec 2.0, with significant reduction of training cost (80% for BASE model, 65% for LARGE model). Furthermore, we address the problem of noise-robustness that is critical to real-world speech applications. We propose multi-condition pre-training by perturbing the student's input with various types of additive noise. We demonstrate that multi-condition pre-trained SPIRAL models are more robust to noisy speech (9.0% - 13.3% relative word error rate reduction on real noisy test data), compared to applying multi-condition training solely in the fine-tuning stage. Source code is available at https://github.com/huawei-noah/Speech-Backbones/tree/main/SPIRAL.