MParrotTTS: Multilingual Multi-speaker Text to Speech Synthesis in Low Resource Setting
This addresses the challenge of generating high-quality speech for multiple languages and speakers with limited data, which is incremental as it builds on self-supervised representations but offers novel capabilities like cross-lingual voice transfer.
The paper tackles the problem of multilingual, multi-speaker text-to-speech synthesis in low-resource settings by introducing MParrotTTS, which adapts to new languages with minimal supervised data and transfers voices across languages without bilingual examples, outperforming state-of-the-art models using only a small fraction of training data.
We present MParrotTTS, a unified multilingual, multi-speaker text-to-speech (TTS) synthesis model that can produce high-quality speech. Benefiting from a modularized training paradigm exploiting self-supervised speech representations, MParrotTTS adapts to a new language with minimal supervised data and generalizes to languages not seen while training the self-supervised backbone. Moreover, without training on any bilingual or parallel examples, MParrotTTS can transfer voices across languages while preserving the speaker-specific characteristics, e.g., synthesizing fluent Hindi speech using a French speaker's voice and accent. We present extensive results on six languages in terms of speech naturalness and speaker similarity in parallel and cross-lingual synthesis. The proposed model outperforms the state-of-the-art multilingual TTS models and baselines, using only a small fraction of supervised training data. Speech samples from our model can be found at https://paper2438.github.io/tts/