Degeneration in VAE: in the Light of Fisher Information Loss
This addresses a key issue in representation learning for VAE users, but it is incremental as it builds on existing skip connection techniques.
The paper tackles the problem of degeneration in Variational Autoencoders (VAE), where deep networks weaken input-latent code correlation, and shows that skip connections (SCVAE) mitigate this by preserving information without architectural changes, improving performance in experiments.
While enormous progress has been made to Variational Autoencoder (VAE) in recent years, similar to other deep networks, VAE with deep networks suffers from the problem of degeneration, which seriously weakens the correlation between the input and the corresponding latent codes, deviating from the goal of the representation learning. To investigate how degeneration affects VAE from a theoretical perspective, we illustrate the information transmission in VAE and analyze the intermediate layers of the encoders/decoders. Specifically, we propose a Fisher Information measure for the layer-wise analysis. With such measure, we demonstrate that information loss is ineluctable in feed-forward networks and causes the degeneration in VAE. We show that skip connections in VAE enable the preservation of information without changing the model architecture. We call this class of VAE equipped with skip connections as SCVAE and perform a range of experiments to show its advantages in information preservation and degeneration mitigation.