Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms
This work addresses the problem of slow and inaccurate inference for researchers and practitioners using discrete diffusion models, representing an incremental improvement over existing approximate methods.
The paper tackles the challenge of inefficient inference in discrete diffusion models by introducing high-order numerical schemes, achieving second-order accuracy and superior sample quality on GPT-2 text and ImageNet image generation tasks.
Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $τ$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $τ$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $θ$-trapezoidal method in KL divergence. Empirical evaluations on GPT-2 level text and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints.