Do Prosody Transfer Models Transfer Prosody?
This work addresses a key inconsistency in prosody transfer models for TTS synthesis, revealing limitations that could impact their practical application in generating natural-sounding speech.
The paper investigates whether prosody transfer models in Text-to-Speech synthesis actually transfer prosody by proposing a training method using prosodically-related but different utterances, and finds that this leads to significantly worse performance compared to using identical utterances, suggesting the models learn utterance-level representations rather than transferable prosody.
Some recent models for Text-to-Speech synthesis aim to transfer the prosody of a reference utterance to the generated target synthetic speech. This is done by using a learned embedding of the reference utterance, which is used to condition speech generation. During training, the reference utterance is identical to the target utterance. Yet, during synthesis, these models are often used to transfer prosody from a reference that differs from the text or speaker being synthesized. To address this inconsistency, we propose to use a different, but prosodically-related, utterance during training too. We believe this should encourage the model to learn to transfer only those characteristics that the reference and target have in common. If prosody transfer methods do indeed transfer prosody they should be able to be trained in the way we propose. However, results show that a model trained under these conditions performs significantly worse than one trained using the target utterance as a reference. To explain this, we hypothesize that prosody transfer models do not learn a transferable representation of prosody, but rather an utterance-level representation which is highly dependent on both the reference speaker and reference text.