Resampled Priors for Variational Autoencoders
This work addresses a specific bottleneck in VAE training for machine learning researchers, offering an incremental improvement over prior methods.
The authors tackled the problem of underfitting in Variational Autoencoders (VAEs) caused by simple priors by proposing Learned Accept/Reject Sampling (LARS) to construct richer priors, resulting in improved performance on standard datasets, including when combined with existing methods for an additional boost.
We propose Learned Accept/Reject Sampling (LARS), a method for constructing richer priors using rejection sampling with a learned acceptance function. This work is motivated by recent analyses of the VAE objective, which pointed out that commonly used simple priors can lead to underfitting. As the distribution induced by LARS involves an intractable normalizing constant, we show how to estimate it and its gradients efficiently. We demonstrate that LARS priors improve VAE performance on several standard datasets both when they are learned jointly with the rest of the model and when they are fitted to a pretrained model. Finally, we show that LARS can be combined with existing methods for defining flexible priors for an additional boost in performance.