Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models
This work addresses the need for efficient deployment of diffusion models, particularly as architectures evolve, but it is incremental as it builds on existing quantization techniques.
The paper tackles the problem of understanding and optimizing quantization sensitivity in diffusion models to reduce inference cost and model size, resulting in a framework that achieves 3.4-bit to 3.9-bit weight quantization on various models and outperforms existing methods with 6-bit activation quantization in metrics and image quality.
Diffusion Models (DM) have democratized AI image generation through an iterative denoising process. Quantization is a major technique to alleviate the inference cost and reduce the size of DM denoiser networks. However, as denoisers evolve from variants of convolutional U-Nets toward newer Transformer architectures, it is of growing importance to understand the quantization sensitivity of different weight layers, operations and architecture types to performance. In this work, we address this challenge with Qua$^2$SeDiMo, a mixed-precision Post-Training Quantization framework that generates explainable insights on the cost-effectiveness of various model weight quantization methods for different denoiser operation types and block structures. We leverage these insights to make high-quality mixed-precision quantization decisions for a myriad of diffusion models ranging from foundational U-Nets to state-of-the-art Transformers. As a result, Qua$^2$SeDiMo can construct 3.4-bit, 3.9-bit, 3.65-bit and 3.7-bit weight quantization on PixArt-$α$, PixArt-$Σ$, Hunyuan-DiT and SDXL, respectively. We further pair our weight-quantization configurations with 6-bit activation quantization and outperform existing approaches in terms of quantitative metrics and generative image quality.