Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck
This addresses a fundamental bottleneck in VAE-based text generation for NLP researchers and practitioners, though it represents an incremental improvement over existing VAE architectures.
The paper tackles the problem of variational autoencoders ignoring latent variables in text generation by introducing a discretized bottleneck that enforces implicit latent feature matching. The approach demonstrates effectiveness across multiple NLP tasks including language modeling, text style transfer, dialog generation, and machine translation.
Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. However, the sequential text generation common pitfall with VAEs is that the model tends to ignore latent variables with a strong auto-regressive decoder. In this paper, we propose a principled approach to alleviate this issue by applying a discretized bottleneck to enforce an implicit latent feature matching in a more compact latent space. We impose a shared discrete latent space where each input is learned to choose a combination of latent atoms as a regularized latent representation. Our model endows a promising capability to model underlying semantics of discrete sequences and thus provide more interpretative latent structures. Empirically, we demonstrate our model's efficiency and effectiveness on a broad range of tasks, including language modeling, unaligned text style transfer, dialog response generation, and neural machine translation.