M3ST: Mix at Three Levels for Speech Translation
This addresses data scarcity for speech translation researchers, offering incremental improvements through novel data augmentation techniques.
The paper tackles data scarcity in end-to-end speech-to-text translation by proposing M3ST, a method that mixes data at word, sentence, and frame levels during fine-tuning, achieving state-of-the-art results with an average BLEU score of 29.9 on eight directions in the MuST-C benchmark.
How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.