Translatotron 2: High-quality direct speech-to-speech translation with voice preservation
This addresses the problem of efficient and privacy-preserving multilingual communication for users needing voice translation, though it builds incrementally on prior work.
The paper tackles direct speech-to-speech translation by introducing Translatotron 2, which improves translation quality by up to +15.5 BLEU over its predecessor and approaches cascade systems, while also preserving speaker voices across languages without requiring segmentation.
We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. Experimental results on three datasets consistently show that Translatotron 2 outperforms the original Translatotron by a large margin on both translation quality (up to +15.5 BLEU) and speech generation quality, and approaches the same of cascade systems. In addition, we propose a simple method for preserving speakers' voices from the source speech to the translation speech in a different language. Unlike existing approaches, the proposed method is able to preserve each speaker's voice on speaker turns without requiring for speaker segmentation. Furthermore, compared to existing approaches, it better preserves speaker's privacy and mitigates potential misuse of voice cloning for creating spoofing audio artifacts.