Beta-VAE has 2 Behaviors: PCA or ICA?
This work provides insights into representation learning dynamics in Beta-VAE, which is incremental for researchers in disentangled representation learning.
The paper investigates how the number of latent variables in Beta-VAE influences learned representations, finding that few variables lead to PCA-like principal components, while many variables result in ICA-like disentangled representations, with the effect attributed to competition for information bandwidth.
Beta-VAE is a very classical model for disentangled representation learning, the use of an expanding bottleneck that allow information into the decoder gradually is key to representation disentanglement as well as high-quality reconstruction. During recent experiments on such fascinating structure, we discovered that the total amount of latent variables can affect the representation learnt by the network: with very few latent variables, the network tend to learn the most important or principal variables, acting like a PCA; with very large numbers of latent variables, the variables tend to be more disentangled, and act like an ICA. Our assumption is that the competition between latent variables while trying to gain the most information bandwidth can lead to this phenomenon.