Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation
For researchers working on efficient generative models, this work reveals a fundamental limitation of distribution matching distillation that undermines its intended flexibility.
The paper identifies and explains 'copying' behavior in few-step distillation of diffusion models, where students reproduce the teacher's noise-data pairings despite distribution-level supervision, and attributes it to limited geometric freedom in high-dimensional settings.
Distribution Matching Distillation (DMD) compresses pretrained diffusion models into efficient few-step generators by aligning their noised distributions across all scales. In principle, such distribution-level supervision remains agnostic to specific noise-data pairings of the teacher; this provides the student the freedom to remap latent noise, a behavior consistently observed in low-dimensional settings. Surprisingly, we find that in high-dimensional settings, distilled students spontaneously reproduce the original noise-data pairings of the teacher, a phenomenon we term copying. We demonstrate that copying is neither a byproduct of adversarial objectives nor a result of teacher memorization. Instead, our evidence suggests that copying is an emergent property arising from the limited geometric freedom of the student model during high-dimensional distillation.