Mutual Information Collapse Explains Disentanglement Failure in $β$-VAEs
This addresses a fundamental issue in unsupervised disentanglement for machine learning, providing a theoretical explanation and practical solution for researchers working with variational autoencoders.
The paper tackles the problem of disentanglement failure in β-VAEs, showing that strong regularization causes a collapse in latent-factor mutual information, and introduces a λβ-VAE that stabilizes disentanglement across a broader range of β values, with experiments confirming improved performance on datasets like dSprites and Shapes3D.
The $β$-VAE is a foundational framework for unsupervised disentanglement, using $β$ to regulate the trade-off between latent factorization and reconstruction fidelity. Empirically, however, disentanglement performance exhibits a pervasive non-monotonic trend: benchmarks such as MIG and SAP typically peak at intermediate $β$ and collapse as regularization increases. We demonstrate that this collapse is a fundamental information-theoretic failure, where strong Kullback-Leibler pressure promotes marginal independence at the expense of the latent channel's semantic informativeness. By formalizing this mechanism in a linear-Gaussian setting, we prove that for $β> 1$, stationarity-induced dynamics trigger a spectral contraction of the encoder gain, driving latent-factor mutual information to zero. To resolve this, we introduce the $λβ$-VAE, which decouples regularization pressure from informational collapse via an auxiliary $L_2$ reconstruction penalty $λ$. Extensive experiments on dSprites, Shapes3D, and MPI3D-real confirm that $λ> 0$ stabilizes disentanglement and restores latent informativeness over a significantly broader range of $β$, providing a principled theoretical justification for dual-parameter regularization in variational inference backbones.