Simple Denoising Diffusion Language Models
This work addresses limitations in diffusion-based language models for NLP researchers, offering incremental improvements in training stability and generation quality.
The authors tackled the problem of diffusion models for language generation degrading in few-step regimes and having complex loss formulations, proposing a simplified denoising loss that stabilizes training and matches ELBO-level performance, with a contrastive-inspired modification yielding additional improvements in generation quality.
Diffusion models have recently been extended to language generation through Masked Diffusion Language Models (MDLMs), which achieve performance competitive with strong autoregressive models. However, MDLMs tend to degrade in the few-step regime and cannot directly adopt existing few-step distillation methods designed for continuous diffusion models, as they lack the intrinsic property of mapping from noise to data. Recent Uniform-state Diffusion Models (USDMs), initialized from a uniform prior, alleviate some limitations but still suffer from complex loss formulations that hinder scalability. In this work, we propose a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training and matching ELBO-level performance. Furthermore, by framing denoising as self-supervised learning, we introduce a simple modification to our denoising loss with contrastive-inspired negative gradients, which is practical and yield additional improvements in generation quality.