Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey
This paper aims to provide a comprehensive educational resource for researchers and practitioners interested in understanding the theoretical underpinnings and practical applications of these related dimensionality reduction and generative models.
This paper provides a tutorial and survey of factor analysis, probabilistic Principal Component Analysis (PCA), variational inference, and Variational Autoencoder (VAE), all of which are dimensionality reduction and generative models. It explains how these methods assume data points are generated from low-dimensional latent factors, enabling both dimensionality reduction and the generation of new data.
This is a tutorial and survey paper on factor analysis, probabilistic Principal Component Analysis (PCA), variational inference, and Variational Autoencoder (VAE). These methods, which are tightly related, are dimensionality reduction and generative models. They assume that every data point is generated from or caused by a low-dimensional latent factor. By learning the parameters of distribution of latent space, the corresponding low-dimensional factors are found for the sake of dimensionality reduction. For their stochastic and generative behaviour, these models can also be used for generation of new data points in the data space. In this paper, we first start with variational inference where we derive the Evidence Lower Bound (ELBO) and Expectation Maximization (EM) for learning the parameters. Then, we introduce factor analysis, derive its joint and marginal distributions, and work out its EM steps. Probabilistic PCA is then explained, as a special case of factor analysis, and its closed-form solutions are derived. Finally, VAE is explained where the encoder, decoder and sampling from the latent space are introduced. Training VAE using both EM and backpropagation are explained.