A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
This addresses a critical instability in deep latent variable modeling of text, offering a simple fix that challenges the adequacy of the standard training objective for balancing representation learning and data modeling.
The paper tackles the problem of posterior collapse in Variational Autoencoders (VAEs) for text modeling by combining two known heuristics, which substantially improves held-out likelihood, reconstruction, and latent representation learning compared to previous state-of-the-art methods, despite obtaining a worse ELBO.
When trained effectively, the Variational Autoencoder (VAE) is both a powerful language model and an effective representation learning framework. In practice, however, VAEs are trained with the evidence lower bound (ELBO) as a surrogate objective to the intractable marginal data likelihood. This approach to training yields unstable results, frequently leading to a disastrous local optimum known as posterior collapse. In this paper, we investigate a simple fix for posterior collapse which yields surprisingly effective results. The combination of two known heuristics, previously considered only in isolation, substantially improves held-out likelihood, reconstruction, and latent representation learning when compared with previous state-of-the-art methods. More interestingly, while our experiments demonstrate superiority on these principle evaluations, our method obtains a worse ELBO. We use these results to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling.