Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
This work addresses efficiency and performance in language modeling for researchers and practitioners, offering a novel perspective that unifies diffusion models with autoregressive approaches, though it is incremental in building upon existing discrete diffusion methods.
The paper tackles the problem of improving discrete diffusion models for language modeling by revealing that the concrete score can be expressed as conditional probabilities of clean data, leading to the proposal of RADD, which achieves state-of-the-art performance on 5 zero-shot language modeling benchmarks at the GPT-2 scale with reduced function evaluations.
Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbing discrete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted as an expected negative log-likelihood for AO-ARMs. Further, our RADD models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at https://github.com/ML-GSAI/RADD.