ReCAB-VAE: Gumbel-Softmax Variational Inference Based on Analytic Divergence
This work addresses a specific bottleneck in variational inference for discrete latent representations, offering an incremental improvement for applications like speech synthesis.
The authors tackled the problem of modeling discrete latent variables in variational autoencoders by proposing a new divergence metric (ReCAB) with a closed-form solution, which improved speech quality in an emotional text-to-speech synthesis system, achieving better performance than baselines.
The Gumbel-softmax distribution, or Concrete distribution, is often used to relax the discrete characteristics of a categorical distribution and enable back-propagation through differentiable reparameterization. Although it reliably yields low variance gradients, it still relies on a stochastic sampling process for optimization. In this work, we present a relaxed categorical analytic bound (ReCAB), a novel divergence-like metric which corresponds to the upper bound of the Kullback-Leibler divergence (KLD) of a relaxed categorical distribution. The proposed metric is easy to implement because it has a closed form solution, and empirical results show that it is close to the actual KLD. Along with this new metric, we propose a relaxed categorical analytic bound variational autoencoder (ReCAB-VAE) that successfully models both continuous and relaxed discrete latent representations. We implement an emotional text-to-speech synthesis system based on the proposed framework, and show that the proposed system flexibly and stably controls emotion expressions with better speech quality compared to baselines that use stochastic estimation or categorical distribution approximation.