A Generalised Linear Model Framework for $β$-Variational Autoencoders based on Exponential Dispersion Families
This work provides incremental theoretical insights for researchers in variational autoencoders, addressing issues like posterior collapse and auto-pruning.
The authors tackled the problem of understanding critical points and training performance in β-VAEs by connecting them to generalized linear models and exponential dispersion families, resulting in a method to initialize networks with maximum likelihood estimates that improved training on synthetic and real-world datasets.
Although variational autoencoders (VAE) are successfully used to obtain meaningful low-dimensional representations for high-dimensional data, the characterization of critical points of the loss function for general observation models is not fully understood. We introduce a theoretical framework that is based on a connection between $β$-VAE and generalized linear models (GLM). The equality between the activation function of a $β$-VAE and the inverse of the link function of a GLM enables us to provide a systematic generalization of the loss analysis for $β$-VAE based on the assumption that the observation model distribution belongs to an exponential dispersion family (EDF). As a result, we can initialize $β$-VAE nets by maximum likelihood estimates (MLE) that enhance the training performance on both synthetic and real world data sets. As a further consequence, we analytically describe the auto-pruning property inherent in the $β$-VAE objective and reason for posterior collapse.