Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice
This improves neural TTS for applications requiring quick speaker adaptation or enhanced voice quality, but it is incremental as it builds on existing multi-speaker TTS methods.
The paper tackles the problem of adapting neural TTS to new speakers with limited data and enhancing premium voices by modeling a multi-speaker latent space, achieving MOS scores of 4.16 in naturalness and 4.64 in speaker similarity with less than 5 minutes of training data, and 4.5 for out-of-domain texts for premium voices.
Neural TTS has shown it can generate high quality synthesized speech. In this paper, we investigate the multi-speaker latent space to improve neural TTS for adapting the system to new speakers with only several minutes of speech or enhancing a premium voice by utilizing the data from other speakers for richer contextual coverage and better generalization. A multi-speaker neural TTS model is built with the embedded speaker information in both spectral and speaker latent space. The experimental results show that, with less than 5 minutes of training data from a new speaker, the new model can achieve an MOS score of 4.16 in naturalness and 4.64 in speaker similarity close to human recordings (4.74). For a well-trained premium voice, we can achieve an MOS score of 4.5 for out-of-domain texts, which is comparable to an MOS of 4.58 for professional recordings, and significantly outperforms single speaker result of 4.28.