Physics Informed Distillation for Diffusion Models
This addresses the computational bottleneck in diffusion models for generative modeling, offering an easy-to-use distillation approach, though it is incremental as it builds on existing distillation and PINN concepts.
The paper tackles the slow image generation of diffusion models by introducing Physics Informed Distillation (PID), which uses a student model to represent the ODE system of a teacher diffusion model, achieving performance comparable to recent distillation methods on CIFAR-10 and ImageNet 64x64 without needing synthetic dataset generation.
Diffusion models have recently emerged as a potent tool in generative modeling. However, their inherent iterative nature often results in sluggish image generation due to the requirement for multiple model evaluations. Recent progress has unveiled the intrinsic link between diffusion models and Probability Flow Ordinary Differential Equations (ODEs), thus enabling us to conceptualize diffusion models as ODE systems. Simultaneously, Physics Informed Neural Networks (PINNs) have substantiated their effectiveness in solving intricate differential equations through implicit modeling of their solutions. Building upon these foundational insights, we introduce Physics Informed Distillation (PID), which employs a student model to represent the solution of the ODE system corresponding to the teacher diffusion model, akin to the principles employed in PINNs. Through experiments on CIFAR 10 and ImageNet 64x64, we observe that PID achieves performance comparable to recent distillation methods. Notably, it demonstrates predictable trends concerning method-specific hyperparameters and eliminates the need for synthetic dataset generation during the distillation process. Both of which contribute to its easy-to-use nature as a distillation approach for Diffusion Models. Our code and pre-trained checkpoint are publicly available at: https://github.com/pantheon5100/pid_diffusion.git.