Auto-Encoding Total Correlation Explanation
This work addresses the interpretability issue in representation learning for researchers and practitioners in machine learning, offering an incremental improvement by relaxing assumptions in existing methods.
The paper tackles the problem of inscrutable representations in unsupervised learning by proposing an information-theoretic approach using total correlation to characterize disentanglement and dependence, resulting in a new algorithm, AnchorVAE, that improves interpretability and sample generation.
Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using multivariate mutual information, also called total correlation. The principle of total Cor-relation Ex-planation (CorEx) has motivated successful unsupervised learning applications across a variety of domains, but under some restrictive assumptions. Here we relax those restrictions by introducing a flexible variational lower bound to CorEx. Surprisingly, we find that this lower bound is equivalent to the one in variational autoencoders (VAE) under certain conditions. This information-theoretic view of VAE deepens our understanding of hierarchical VAE and motivates a new algorithm, AnchorVAE, that makes latent codes more interpretable through information maximization and enables generation of richer and more realistic samples.