A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
This work addresses the problem of expensive labeled speech data for ASR and ST researchers, offering an incremental improvement by integrating text data to enhance model performance.
The paper tackles the challenge of limited labeled speech data for automatic speech recognition (ASR) and speech translation (ST) by proposing a multi-task learning framework that leverages text data through auxiliary tasks, achieving a 10-15% relative word error rate reduction on Librispeech and 3.6-9.2 BLEU improvement on MuST-C tasks.
Attention-based sequence-to-sequence modeling provides a powerful and elegant solution for applications that need to map one sequence to a different sequence. Its success heavily relies on the availability of large amounts of training data. This presents a challenge for speech applications where labelled speech data is very expensive to obtain, such as automatic speech recognition (ASR) and speech translation (ST). In this study, we propose a general multi-task learning framework to leverage text data for ASR and ST tasks. Two auxiliary tasks, a denoising autoencoder task and machine translation task, are proposed to be co-trained with ASR and ST tasks respectively. We demonstrate that representing text input as phoneme sequences can reduce the difference between speech and text inputs, and enhance the knowledge transfer from text corpora to the speech to text tasks. Our experiments show that the proposed method achieves a relative 10~15% word error rate reduction on the English Librispeech task compared with our baseline, and improves the speech translation quality on the MuST-C tasks by 3.6~9.2 BLEU.