Improving multi-speaker TTS prosody variance with a residual encoder and normalizing flows
This work addresses the issue of low expressiveness in TTS systems for applications requiring natural and variable speech synthesis, representing an incremental improvement over existing methods.
The paper tackled the problem of flat prosody in multi-speaker text-to-speech systems by proposing a new model that replaces the reference encoder with a learned latent distribution for prosody and uses flow-normalized speaker embeddings, resulting in significantly higher prosody variance and speaker distinctiveness without reducing intelligibility.
Text-to-speech systems recently achieved almost indistinguishable quality from human speech. However, the prosody of those systems is generally flatter than natural speech, producing samples with low expressiveness. Disentanglement of speaker id and prosody is crucial in text-to-speech systems to improve on naturalness and produce more variable syntheses. This paper proposes a new neural text-to-speech model that approaches the disentanglement problem by conditioning a Tacotron2-like architecture on flow-normalized speaker embeddings, and by substituting the reference encoder with a new learned latent distribution responsible for modeling the intra-sentence variability due to the prosody. By removing the reference encoder dependency, the speaker-leakage problem typically happening in this kind of systems disappears, producing more distinctive syntheses at inference time. The new model achieves significantly higher prosody variance than the baseline in a set of quantitative prosody features, as well as higher speaker distinctiveness, without decreasing the speaker intelligibility. Finally, we observe that the normalized speaker embeddings enable much richer speaker interpolations, substantially improving the distinctiveness of the new interpolated speakers.