Learning to Integrate Diffusion ODEs by Averaging the Derivatives
This addresses the need for faster and more stable inference in diffusion models, offering an intermediate solution between numerical solvers and distillation techniques, with incremental improvements in specific domains like image generation.
This work tackles the problem of accelerating diffusion model inference by introducing secant losses, which balance performance and cost through learning ODE integration, achieving a 10-step FID of 2.14 on CIFAR-10 and a 4-step FID of 2.27 on ImageNet-256.
To accelerate diffusion model inference, numerical solvers perform poorly at extremely small steps, while distillation techniques often introduce complexity and instability. This work presents an intermediate strategy, balancing performance and cost, by learning ODE integration using loss functions derived from the derivative-integral relationship, inspired by Monte Carlo integration and Picard iteration. From a geometric perspective, the losses operate by gradually extending the tangent to the secant, thus are named as secant losses. The secant losses can rapidly convert (via fine-tuning or distillation) a pretrained diffusion model into its secant version. In our experiments, the secant version of EDM achieves a $10$-step FID of $2.14$ on CIFAR-10, while the secant version of SiT-XL/2 attains a $4$-step FID of $2.27$ and an $8$-step FID of $1.96$ on ImageNet-$256\times256$. Code will be available.