Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network
For practitioners deploying diffusion models, this provides a practical speedup without quality loss, though it is an incremental improvement over existing acceleration methods.
Dual-Rate Diffusion accelerates diffusion model sampling by interleaving a heavy context encoder (evaluated sparsely) with a light denoising model (evaluated at every step), reducing computational cost by 2-4x on ImageNet while matching standard baseline performance.
Diffusion models achieve state-of-the-art generative performance but suffer from high computational costs during inference due to the repeated evaluation of a heavy neural network. In this work, we propose Dual-Rate Diffusion, a method to accelerate sampling by interleaving the execution of a heavy high-capacity context encoder and a light efficient denoising model. The context encoder is evaluated sparsely to extract high-dimensional features, which are effectively reused by the light denoising model at every step to refine the sample efficiently. This approach significantly accelerates inference without compromising sample quality. On ImageNet benchmarks, Dual-Rate Diffusion matches the performance of standard baselines while reducing computational cost by a factor of $2$-$4$. Furthermore, we demonstrate that our method is compatible with distillation techniques, such as Moment Matching Distillation, enabling further efficiency gains in few-step generation.