Semi-Supervised Generative Modeling for Controllable Speech Synthesis
This work addresses the challenge of interpretable and controllable speech synthesis for applications requiring fine-grained audio generation, representing an incremental improvement over existing unsupervised methods.
The paper tackles the problem of controlling rarely labeled speech attributes like affect and speaking rate in text-to-speech synthesis by introducing a semi-supervised generative model that uses partial supervision on latent variables, achieving reliable control with as little as 1% supervision (30 minutes) without degrading synthesis quality compared to state-of-the-art baselines.
We present a novel generative model that combines state-of-the-art neural text-to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. Audio samples are available on the web.