Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
This addresses the challenge of limited paired speech-text data for ASR systems, offering an incremental improvement in model performance for speech recognition applications.
This paper tackles the problem of improving automatic speech recognition (ASR) by pre-training encoder-decoder models with unpaired speech data, using a novel technique called Speech2C that introduces two pre-training tasks with acoustic units. The result is a relative reduction in word error rate (WER) by 19.2% over methods without decoder pre-training and outperforms state-of-the-art models like wav2vec 2.0 and HuBERT on fine-tuning subsets.
This paper studies a novel pre-training technique with unpaired speech data, Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within a multi-task learning framework, we introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes, derived from an offline clustering model. One is to predict the pseudo codes via masked language modeling in encoder output, like HuBERT model, while the other lets the decoder learn to reconstruct pseudo codes autoregressively instead of generating textual scripts. In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text. Comprehensive experiments on the LibriSpeech corpus show that the proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training, and also outperforms significantly the state-of-the-art wav2vec 2.0 and HuBERT on fine-tuning subsets of 10h and 100h. We release our code and model at https://github.com/microsoft/SpeechT5/tree/main/Speech2C.