nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speech
This addresses the problem of practical multi-speaker TTS applications where limited data is available, though it is incremental as it builds on conditional variational autoencoder frameworks.
The authors tackled the challenge of multi-speaker text-to-speech with minimal adaptation data by proposing nnSpeech, a zero-shot method that synthesizes new speaker voices using only one adaptation utterance without fine-tuning, achieving natural and similar speech across English and Mandarin corpora.
Multi-speaker text-to-speech (TTS) using a few adaption data is a challenge in practical applications. To address that, we propose a zero-shot multi-speaker TTS, named nnSpeech, that could synthesis a new speaker voice without fine-tuning and using only one adaption utterance. Compared with using a speaker representation module to extract the characteristics of new speakers, our method bases on a speaker-guided conditional variational autoencoder and can generate a variable Z, which contains both speaker characteristics and content information. The latent variable Z distribution is approximated by another variable conditioned on reference mel-spectrogram and phoneme. Experiments on the English corpus, Mandarin corpus, and cross-dataset proves that our model could generate natural and similar speech with only one adaption speech.