Continuous Diffusion Model for Language Modeling
This addresses the performance gap in diffusion models for discrete data like language, which is an incremental improvement over existing methods.
The authors tackled the problem of diffusion models underperforming on discrete data like language by proposing a continuous diffusion model that incorporates categorical distribution geometry, establishing a connection between discrete diffusion and continuous flow on statistical manifolds. Their method outperforms existing discrete diffusion models and approaches autoregressive model performance on language modeling benchmarks.
Diffusion models have emerged as a promising alternative to autoregressive models in modeling discrete categorical data. However, diffusion models that directly work on discrete data space fail to fully exploit the power of iterative refinement, as the signals are lost during transitions between discrete states. Existing continuous diffusion models for discrete data underperform compared to discrete methods, and the lack of a clear connection between the two approaches hinders the development of effective diffusion models for discrete data. In this work, we propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution. We establish a connection between the discrete diffusion and continuous flow on the statistical manifold, and building on this analogy, introduce a simple diffusion process that generalizes existing discrete diffusion models. We further propose a simulation-free training framework based on radial symmetry, along with a simple technique to address the high dimensionality of the manifold. Comprehensive experiments on language modeling benchmarks and other modalities show that our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models. The code is available at https://github.com/harryjo97/RDLM.