Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
This work addresses interpretable prosody modeling for text-to-speech synthesis, representing an incremental improvement with specific gains in disentanglement.
The paper tackles the problem of modeling fine-grained prosody in speech synthesis by proposing a hierarchical latent variable model based on Tacotron 2, achieving multi-resolution prosody modeling without degrading reconstruction performance while improving interpretability and disentanglement of latent dimensions.
This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution modeling of prosody by conditioning finer level representations on coarser level ones. Additionally, it imposes hierarchical conditioning across all latent dimensions using a conditional variational auto-encoder (VAE) with an auto-regressive structure. Evaluation of reconstruction performance illustrates that the new structure does not degrade the model while allowing better interpretability. Interpretations of prosody attributes are provided together with the comparison between word-level and phone-level prosody representations. Moreover, both qualitative and quantitative evaluations are used to demonstrate the improvement in the disentanglement of the latent dimensions.