Tackling Over-pruning in Variational Autoencoders
This addresses a specific bottleneck in variational autoencoders for researchers and practitioners in generative modeling, though it is incremental as it builds on existing VAE frameworks.
The paper tackles the problem of factor over-pruning in variational autoencoders, which limits modeling power and latent representation diversity, by proposing the epitomic variational autoencoder (eVAE) that uses groups of competing latent factors to prevent inactive units, resulting in more efficient model capacity use and better generalization than standard VAE on MNIST and TFD datasets.
Variational autoencoders (VAE) are directed generative models that learn factorial latent variables. As noted by Burda et al. (2015), these models exhibit the problem of factor over-pruning where a significant number of stochastic factors fail to learn anything and become inactive. This can limit their modeling power and their ability to learn diverse and meaningful latent representations. In this paper, we evaluate several methods to address this problem and propose a more effective model-based approach called the epitomic variational autoencoder (eVAE). The so-called epitomes of this model are groups of mutually exclusive latent factors that compete to explain the data. This approach helps prevent inactive units since each group is pressured to explain the data. We compare the approaches with qualitative and quantitative results on MNIST and TFD datasets. Our results show that eVAE makes efficient use of model capacity and generalizes better than VAE.