Multi-Task Adversarial Training Algorithm for Multi-Speaker Neural Text-to-Speech
This addresses the generalization issue in multi-speaker TTS for applications requiring voice synthesis of new speakers, though it is incremental as it builds on existing GAN frameworks.
The paper tackles the problem of synthesizing high-quality speech for unseen speakers in multi-speaker neural text-to-speech by proposing a multi-task adversarial training algorithm, which improves synthetic speech quality over a conventional GAN-based method as shown in experimental evaluation.
We propose a novel training algorithm for a multi-speaker neural text-to-speech (TTS) model based on multi-task adversarial training. A conventional generative adversarial network (GAN)-based training algorithm significantly improves the quality of synthetic speech by reducing the statistical difference between natural and synthetic speech. However, the algorithm does not guarantee the generalization performance of the trained TTS model in synthesizing voices of unseen speakers who are not included in the training data. Our algorithm alternatively trains two deep neural networks: multi-task discriminator and multi-speaker neural TTS model (i.e., generator of GANs). The discriminator is trained not only to distinguish between natural and synthetic speech but also to verify the speaker of input speech is existent or non-existent (i.e., newly generated by interpolating seen speakers' embedding vectors). Meanwhile, the generator is trained to minimize the weighted sum of the speech reconstruction loss and adversarial loss for fooling the discriminator, which achieves high-quality multi-speaker TTS even if the target speaker is unseen. Experimental evaluation shows that our algorithm improves the quality of synthetic speech better than a conventional GANSpeech algorithm.