Multilingual Speech-to-Speech Translation into Multiple Target Languages
This work addresses a significant gap in multilingual speech-to-speech translation by enabling multiple target languages, which is incremental but important for improving cross-lingual spoken communication.
The paper tackles the problem of speech-to-speech translation into multiple target languages, presenting the first model that supports this capability and showing superior performance over bilingual models in translating English into 16 target languages.
Speech-to-speech translation (S2ST) enables spoken communication between people talking in different languages. Despite a few studies on multilingual S2ST, their focus is the multilinguality on the source side, i.e., the translation from multiple source languages to one target language. We present the first work on multilingual S2ST supporting multiple target languages. Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we equip these key components with multilingual capability. Speech-to-masked-unit (S2MU) is the multilingual extension of S2U, which applies masking to units which don't belong to the given target language to reduce the language interference. We also propose multilingual vocoder which is trained with language embedding and the auxiliary loss of language identification. On benchmark translation testsets, our proposed multilingual model shows superior performance than bilingual models in the translation from English into $16$ target languages.