Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis
This work addresses a critical issue in TTS synthesis for applications requiring high-quality speech generation, though it is incremental as it builds on existing GAN and flow-based methods.
The authors tackled the mismatch problem between acoustic models and vocoders in two-stage text-to-speech (TTS) systems by proposing Glow-WaveGAN, which uses a GAN-based variational auto-encoder to learn a latent representation directly from speech and a flow-based acoustic model to model this representation from text, resulting in high-fidelity speech that outperforms state-of-the-art GAN-based models.
Current two-stage TTS framework typically integrates an acoustic model with a vocoder -- the acoustic model predicts a low resolution intermediate representation such as Mel-spectrum while the vocoder generates waveform from the intermediate representation. Although the intermediate representation is served as a bridge, there still exists critical mismatch between the acoustic model and the vocoder as they are commonly separately learned and work on different distributions of representation, leading to inevitable artifacts in the synthesized speech. In this work, different from using pre-designed intermediate representation in most previous studies, we propose to use VAE combining with GAN to learn a latent representation directly from speech and then utilize a flow-based acoustic model to model the distribution of the latent representation from text. In this way, the mismatch problem is migrated as the two stages work on the same distribution. Results demonstrate that the flow-based acoustic model can exactly model the distribution of our learned speech representation and the proposed TTS framework, namely Glow-WaveGAN, can produce high fidelity speech outperforming the state-of-the-art GAN-based model.