Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
This addresses text generation for NLP applications, but is incremental as it adapts diffusion models to language tasks.
The authors tackled text generation by introducing GENIE, a diffusion language model pre-trained with a continuous paragraph denoise objective, which achieved comparable performance to state-of-the-art autoregressive models on four benchmarks while generating more diverse text.
In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE. GENIE is a large-scale pretrained diffusion language model that consists of an encoder and a diffusion-based decoder, which can generate text by gradually transforming a random noise sequence into a coherent text sequence. To pre-train GENIE on a large-scale language corpus, we design a new continuous paragraph denoise objective, which encourages the diffusion-decoder to reconstruct a clean text paragraph from a corrupted version, while preserving the semantic and syntactic coherence. We evaluate GENIE on four downstream text generation benchmarks, namely XSum, CNN/DailyMail, Gigaword, and CommonGen. Our experimental results show that GENIE achieves comparable performance with the state-of-the-art autoregressive models on these benchmarks, and generates more diverse text samples. The code and models of GENIE are available at https://github.com/microsoft/ProphetNet/tree/master/GENIE.