A Tutorial on VAEs: From Bayes' Rule to Lossless Compression
It clarifies foundational concepts for researchers and practitioners using VAEs, but is incremental as it synthesizes existing knowledge without new results.
This tutorial provides an overview of Variational Auto-Encoders (VAEs), explaining their derivation from probabilistic and information-theoretic perspectives, and addresses common misconceptions with practical examples.
The Variational Auto-Encoder (VAE) is a simple, efficient, and popular deep maximum likelihood model. Though usage of VAEs is widespread, the derivation of the VAE is not as widely understood. In this tutorial, we will provide an overview of the VAE and a tour through various derivations and interpretations of the VAE objective. From a probabilistic standpoint, we will examine the VAE through the lens of Bayes' Rule, importance sampling, and the change-of-variables formula. From an information theoretic standpoint, we will examine the VAE through the lens of lossless compression and transmission through a noisy channel. We will then identify two common misconceptions over the VAE formulation and their practical consequences. Finally, we will visualize the capabilities and limitations of VAEs using a code example (with an accompanying Jupyter notebook) on toy 2D data.