DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
This addresses the problem of gradient-based training for discrete latent models in machine learning, offering an incremental improvement over existing methods.
The paper tackles the challenge of training discrete latent variable models by proposing overlapping transformations, a new smoothing method based on a mixture of two overlapping distributions, and shows that DVAE++ outperforms other continuous relaxations like Gumbel-Softmax on several benchmarks.
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).