Improving speech translation by fusing speech and text
This work addresses the challenge of integrating disparate modalities for researchers and practitioners in speech and machine translation, offering incremental improvements over existing methods.
The paper tackles the problem of modality gaps in speech translation by proposing FST, a cross-modal model that fuses speech and text inputs, achieving an average LEU of 34.0 on MuST-C En→De/Es/Fr, which is +1.1 BLEU over the state-of-the-art, and improving machine translation by an average of 3.2 BLEU over a pre-trained model.
In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text, which are disparate modalities. We observe three levels of modality gap between them, denoted by Modal input representation, Modal semantic, and Modal hidden states. To tackle these gaps, we propose \textbf{F}use-\textbf{S}peech-\textbf{T}ext (\textbf{FST}), a cross-modal model which supports three distinct input modalities for translation: speech, text, and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation, and fused speech-text translation. We evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that the proposed FST achieves an average 34.0 BLEU on MuST-C En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FST does not degrade on MT task, as observed in prior works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model.