Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
This addresses the problem of limited parallel data for speech translation, offering a novel multimodal approach that is incremental over existing unimodal methods.
The paper tackled the lack of unified representations for speech and text by proposing FAT-MLM, which learns from multimodal data and improves speech translation quality by up to +5.9 BLEU.
Recently, representation learning for text and speech has successfully improved many language related tasks. However, all existing methods suffer from two limitations: (a) they only learn from one input modality, while a unified representation for both speech and text is needed by tasks such as end-to-end speech translation, and as a result,(b) they can not exploit various large-scale text and speech data and their performance is limited by the scarcity of parallel speech translation data.To address these problems, we propose a Fused Acoustic and Text Masked Language Model (FAT-MLM) which jointly learns a unified representation for both acoustic and text input from various types of corpora including parallel data for speech recognition and machine translation, and even pure speech and text data. Within this cross-modal representation learning framework, we further present an end-to-end model for Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three translation directions show that by fine-tuning from FAT-MLM, our proposed speech translation models substantially improve translation quality by up to +5.9 BLEU.