Enhancing Speech-to-Speech Translation with Multiple TTS Targets
This work addresses data scarcity for speech-to-speech translation researchers, offering an incremental improvement through multi-target optimization.
The paper tackles the data scarcity problem in direct speech-to-speech translation by analyzing how synthesized target speech from different TTS systems affects performance, and proposes a multi-task framework that improves S2ST by 2.8 BLEU on the Fisher Spanish-English dataset.
It has been known that direct speech-to-speech translation (S2ST) models usually suffer from the data scarcity issue because of the limited existing parallel materials for both source and target speech. Therefore to train a direct S2ST system, previous works usually utilize text-to-speech (TTS) systems to generate samples in the target language by augmenting the data from speech-to-text translation (S2TT). However, there is a limited investigation into how the synthesized target speech would affect the S2ST models. In this work, we analyze the effect of changing synthesized target speech for direct S2ST models. We find that simply combining the target speech from different TTS systems can potentially improve the S2ST performances. Following that, we also propose a multi-task framework that jointly optimizes the S2ST system with multiple targets from different TTS systems. Extensive experiments demonstrate that our proposed framework achieves consistent improvements (2.8 BLEU) over the baselines on the Fisher Spanish-English dataset.