PixelVAE: A Latent Variable Model for Natural Images
This addresses the problem of capturing both global structure and fine details in unsupervised image modeling for machine learning researchers, though it is incremental as it hybridizes existing methods.
The paper tackled the challenge of natural image modeling by combining Variational Autoencoders (VAEs) and PixelCNN into PixelVAE, achieving state-of-the-art performance on binarized MNIST, competitive results on 64x64 ImageNet, and high-quality samples on LSUN bedrooms.
Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.