Disentangling Style and Speaker Attributes for TTS Style Transfer
This work addresses style transfer challenges in TTS for applications like voice synthesis, but it is incremental as it builds on existing neural TTS methods with specific enhancements.
The paper tackles the problem of limited training data and performance degradation in text-to-speech style transfer, especially for unseen speakers and styles, by proposing a new approach using inverse autoregressive flow and a speaker encoder with six training objectives, achieving superior and more robust performance compared to four prior systems.
End-to-end neural TTS has shown improved performance in speech style transfer. However, the improvement is still limited by the available training data in both target styles and speakers. Additionally, degenerated performance is observed when the trained TTS tries to transfer the speech to a target style from a new speaker with an unknown, arbitrary style. In this paper, we propose a new approach to seen and unseen style transfer training on disjoint, multi-style datasets, i.e., datasets of different styles are recorded, one individual style by one speaker in multiple utterances. An inverse autoregressive flow (IAF) technique is first introduced to improve the variational inference for learning an expressive style representation. A speaker encoder network is then developed for learning a discriminative speaker embedding, which is jointly trained with the rest neural TTS modules. The proposed approach of seen and unseen style transfer is effectively trained with six specifically-designed objectives: reconstruction loss, adversarial loss, style distortion loss, cycle consistency loss, style classification loss, and speaker classification loss. Experiments demonstrate, both objectively and subjectively, the effectiveness of the proposed approach for seen and unseen style transfer tasks. The performance of our approach is superior to and more robust than those of four other reference systems of prior art.