Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models
For practitioners using VAEs, this provides a more general tool to diagnose latent collapse, though the improvement is incremental over existing criteria.
The paper proposes an entropy-based criterion to classify latent dimensions in VAEs as active, passive, or mixed, overcoming the Gaussian prior limitation of existing methods. The criterion consistently recovers the polarized regime across several VAE variants and shows that passive dimensions can still contribute to downstream tasks when properly normalized.
Variational Autoencoders (VAEs) often exhibit a polarised regime in which latent variables separate into active, passive, and mixed subsets. Existing criteria for identifying active dimensions depend on a Gaussian prior, limiting their applicability to variational models and specific priors. We propose a simple information-theoretic classification of the polarised regime based on the entropy of the mean representation. We show theoretically how this entropy couples to KL minimisation through entropy--variance bounds, and we relate the resulting criterion to Bonheme's active/passive conditions. We also clarify a key limitation: entropy of the mean alone cannot reliably distinguish active from mixed dimensions without additional signals from the variance representation. Empirically, we evaluate the entropy criterion on $β$-VAEs, identifiable VAEs, Least-Volume Autoencoders, and L2-regularised autoencoders, and find that it consistently recovers a polarised regime when such a regime is present across the model classes studied. Finally, we show that passive dimensions can yield small but consistent improvements on downstream tasks when latent codes are appropriately normalised, suggesting that collapse is often a matter of scale rather than absolute information removal.