SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity
This work addresses the computational bottleneck in diffusion models for users needing faster and more efficient image generation, representing an incremental improvement through hardware-software co-design.
The paper tackles the slow inference speed of diffusion models for image generation by proposing aggressive quantization and temporal sparsity, resulting in a 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators.
Diffusion models have gained significant popularity in image generation tasks. However, generating high-quality content remains notably slow because it requires running model inference over many time steps. To accelerate these models, we propose to aggressively quantize both weights and activations, while simultaneously promoting significant activation sparsity. We further observe that the stated sparsity pattern varies among different channels and evolves across time steps. To support this quantization and sparsity scheme, we present a novel diffusion model accelerator featuring a heterogeneous mixed-precision dense-sparse architecture, channel-last address mapping, and a time-step-aware sparsity detector for efficient handling of the sparsity pattern. Our 4-bit quantization technique demonstrates superior generation quality compared to existing 4-bit methods. Our custom accelerator achieves 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators.