Improving latent variable descriptiveness with AutoGen
This work addresses a specific bottleneck in variational autoencoders for researchers and practitioners in generative modeling, offering an incremental improvement by formalizing existing training techniques.
The paper tackles the problem of poor latent variable descriptiveness in generative models by proposing an alternative objective that combines data log likelihood with perfect reconstruction likelihood, ensuring the latent variable captures information about observations while maintaining good generation ability. The result shows that this approach yields a lower bound identical to the standard VAE bound with an added pre-factor, providing a formal interpretation for commonly used ad-hoc pre-factors.
Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that combines the data log likelihood as well as the likelihood of a perfect reconstruction through an autoencoder. Tying these together ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, though this approach is a priori unrelated to VAEs, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.