Gacs-Korner Common Information Variational Autoencoder
This work addresses the challenge of disentangling information in machine learning, which is incremental as it builds on existing common information theory and variational autoencoders.
The paper tackles the problem of quantifying and separating shared versus unique information between two random variables by proposing a new notion of common information, which recovers Gács-Körner common information and can be approximated empirically. They demonstrate that their method, using a modified variational autoencoder, learns semantically meaningful factors on high-dimensional data and accurately quantifies common information where ground-truth latents are known.
We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each. Our notion of common information is defined by an optimization problem over a family of functions and recovers the Gács-Körner common information as a special case. Importantly, our notion can be approximated empirically using samples from the underlying data distribution. We then provide a method to partition and quantify the common and unique information using a simple modification of a traditional variational auto-encoder. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos. Moreover, on datasets where ground-truth latent factors are known, we show that we can accurately quantify the common information between the random variables.