Variance Loss in Variational Autoencoders
This addresses a problem for researchers and practitioners using VAEs in generative modeling, particularly in multi-stage settings, but is incremental as it builds on known VAE limitations.
The paper identifies a major issue in Variational Autoencoders (VAEs) where generated data has significantly lower variance than training data, degrading evaluation scores like FID, and shows that renormalizing outputs in a two-stage VAE setting improves sample quality with a sudden burst in FID.
In this article, we highlight what appears to be major issue of Variational Autoencoders, evinced from an extensive experimentation with different network architectures and datasets: the variance of generated data is significantly lower than that of training data. Since generative models are usually evaluated with metrics such as the Frechet Inception Distance (FID) that compare the distributions of (features of) real versus generated images, the variance loss typically results in degraded scores. This problem is particularly relevant in a two stage setting, where we use a second VAE to sample in the latent space of the first VAE. The minor variance creates a mismatch between the actual distribution of latent variables and those generated by the second VAE, that hinders the beneficial effects of the second stage. Renormalizing the output of the second VAE towards the expected normal spherical distribution, we obtain a sudden burst in the quality of generated samples, as also testified in terms of FID.