Conditional [MASK] Discrete Diffusion Language Model
This addresses text generation challenges for NLP applications, but it is incremental as it builds on existing diffusion and masked language model approaches.
The paper tackled the problem of limited diversity and controllability in auto-regressive models for text generation by proposing Diffusion-EAGS, a framework integrating conditional masked language models into diffusion models, which achieved the best quality-diversity tradeoff in experiments.
Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropy-based noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.