Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
This addresses the problem of inefficient quantization-based models for speech synthesis, offering a more efficient alternative.
The paper tackles speech synthesis by proposing a continuous autoregressive model with stochastic monotonic alignment, which outperforms the state-of-the-art VALL-E in evaluations while using only 10.3% of its parameters.
We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the conditional probability distribution. Unlike previous methods that rely on residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of-the-art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3\% of VALL-E's parameters. This demonstrates the potential of continuous speech language models as a more efficient alternative to existing quantization-based speech language models. Sample audio can be found at https://tinyurl.com/gmm-lm-tts.