Speaker verification-derived loss and data augmentation for DNN-based multispeaker speech synthesis
This work addresses a domain-specific problem for speech synthesis researchers and practitioners, offering incremental improvements in handling data scarcity.
The paper tackled the problem of training multispeaker text-to-speech synthesis systems with limited data per speaker, which leads to poor speaker similarity and naturalness. They introduced an additional loss term and data augmentation, showing efficiency in improving speaker similarity and intelligibility through objective and subjective evaluations.
Building multispeaker neural network-based text-to-speech synthesis systems commonly relies on the availability of large amounts of high quality recordings from each speaker and conditioning the training process on the speaker's identity or on a learned representation of it. However, when little data is available from each speaker, or the number of speakers is limited, the multispeaker TTS can be hard to train and will result in poor speaker similarity and naturalness. In order to address this issue, we explore two directions: forcing the network to learn a better speaker identity representation by appending an additional loss term; and augmenting the input data pertaining to each speaker using waveform manipulation methods. We show that both methods are efficient when evaluated with both objective and subjective measures. The additional loss term aids the speaker similarity, while the data augmentation improves the intelligibility of the multispeaker TTS system.