ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs
This addresses prosody modeling for speech synthesis tasks, but it is incremental as it builds on existing style encoders.
The paper tackles the problem of predicting prosodic features like F0 and energy from text and acoustic inputs, showing consistent improvements over state-of-the-art methods and higher prosody preference in TTS perceptual tests.
Prosody conveys rich emotional and semantic information of the speech signal as well as individual idiosyncrasies. We propose a stand-alone model that maps text-to-prosodic features such as F0 and energy and can be used in downstream tasks such as TTS. The ProMode encoder takes as input acoustic features and time-aligned textual content, both are partially masked, and obtains a fixed-length latent prosodic embedding. The decoder predicts acoustics in the masked region using both the encoded prosody input and unmasked textual content. Trained on the GigaSpeech dataset, we compare our method with state-of-the-art style encoders. For F0 and energy predictions, we show consistent improvements for our model at different levels of granularity. We also integrate these predicted prosodic features into a TTS system and conduct perceptual tests, which show higher prosody preference compared to the baselines, demonstrating the model's potential in tasks where prosody modeling is important.