Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
This addresses a bottleneck in discrete diffusion models for researchers and practitioners, enabling faster sampling while preserving performance.
The paper tackles the problem of distilling discrete diffusion models, which is currently difficult, by introducing Discrete Moment Matching Distillation (D-MMD) that maintains high quality and diversity and can outperform teachers on text and image datasets.
It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.