The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
This refines understanding of posterior collapse in VAEs, an important issue for researchers developing generative models, though it is incremental to existing explanations.
The paper challenges the conventional explanation that KL-divergence regularization is primarily responsible for posterior collapse in VAEs, arguing instead that bad local minima in deep autoencoder loss surfaces are a key factor. It proves that small nonlinear perturbations in decoder models can create such minima, and in deeper models, these minima can force VAEs to discard information across all latent dimensions.
In narrow asymptotic settings Gaussian VAE models of continuous data have been shown to possess global optima aligned with ground-truth distributions. Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice. However, contrary to conventional wisdom that largely assigns blame for this phenomena on the undue influence of KL-divergence regularization, we will argue that posterior collapse is, at least in part, a direct consequence of bad local minima inherent to the loss surface of deep autoencoder networks. In particular, we prove that even small nonlinear perturbations of affine VAE decoder models can produce such minima, and in deeper models, analogous minima can force the VAE to behave like an aggressive truncation operator, provably discarding information along all latent dimensions in certain circumstances. Regardless, the underlying message here is not meant to undercut valuable existing explanations of posterior collapse, but rather, to refine the discussion and elucidate alternative risk factors that may have been previously underappreciated.