Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
This work addresses text generation challenges for natural language processing applications, representing an incremental improvement over existing VAE methods.
The paper tackles the problem of generating long and coherent text by proposing multi-level latent variable models based on variational autoencoders, resulting in improved coherence and reduced repetitiveness compared to standard VAEs, with empirical evidence showing mitigation of posterior-collapse issues.
Variational autoencoders (VAEs) have received much attention recently as an end-to-end architecture for text generation with latent variables. In this paper, we investigate several multi-level structures to learn a VAE model to generate long, and coherent text. In particular, we use a hierarchy of stochastic layers between the encoder and decoder networks to generate more informative latent codes. We also investigate a multi-level decoder structure to learn a coherent long-term structure by generating intermediate sentence representations as high-level plan vectors. Empirical results demonstrate that a multi-level VAE model produces more coherent and less repetitive long text compared to the standard VAE models and can further mitigate the posterior-collapse issue.