Pushing the Limits of Zero-shot End-to-End Speech Translation
This addresses data scarcity and modality gaps in speech translation for multilingual applications, representing a significant advance rather than an incremental improvement.
The paper tackles the problem of zero-shot end-to-end speech translation without paired speech-translation data by introducing ZeroSwot, which uses CTC compression and Optimal Transport to align a speech encoder with a multilingual MT model. The method achieves state-of-the-art results on MuST-C and CoVoST benchmarks, outperforming both previous zero-shot and supervised models.
Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MuST-C and CoVoST demonstrate our method's superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.