Distillation of Discrete Diffusion by Exact Conditional Distribution Matching
This work addresses the computational inefficiency of discrete diffusion models for categorical data, offering a more efficient inference method that could benefit applications in generative modeling.
The authors tackled the high inference cost of discrete diffusion models by proposing a distillation method based on exact conditional distribution matching, which reduces the number of function evaluations needed for sampling without relying on approximate simulators or proxy objectives.
Discrete diffusion models (DDMs) are a powerful class of generative models for categorical data, but they typically require many function evaluations for a single sample, making inference expensive. Existing acceleration methods either rely on approximate simulators, such as $τ$-leaping, or on distillation schemes that train new student models and auxiliary networks with proxy objectives. We propose a simple and principled distillation alternative based on \emph{conditional distribution matching}. Our key observation is that the reverse conditional distribution of clean data given a noisy state, $p_{0\mid t}(x_0 \mid x_t)$, admits a Markov decomposition through intermediate times and can be recovered from marginal density ratios and the known forward CTMC kernel. We exploit this structure to define distillation objectives that directly match conditional distributions between a pre-trained teacher and a low-NFE student, both for one-step and few-step samplers.