HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models
This work addresses deployment challenges for diffusion models in resource-constrained environments by enhancing low-bit quantization performance, though it is incremental as it builds on existing quantization methods.
The paper tackles the problem of performance degradation in low-bit quantized diffusion models due to activation outliers, proposing HQ-DM, a quantization-aware training framework that uses Single Hadamard Transformation to reduce these outliers. For ImageNet 256x256 generation with LDM-4, it improves Inception Score by 12.8% for W4A4 and 467.73% for W4A3 over the state-of-the-art.
Diffusion models have demonstrated significant applications in the field of image generation. However, their high computational and memory costs pose challenges for deployment. Model quantization has emerged as a promising solution to reduce storage overhead and accelerate inference. Nevertheless, existing quantization methods for diffusion models struggle to mitigate outliers in activation matrices during inference, leading to substantial performance degradation under low-bit quantization scenarios. To address this, we propose HQ-DM, a novel Quantization-Aware Training framework that applies Single Hadamard Transformation to activation matrices. This approach effectively reduces activation outliers while preserving model performance under quantization. Compared to traditional Double Hadamard Transformation, our proposed scheme offers distinct advantages by seamlessly supporting INT convolution operations while preventing the amplification of weight outliers. For conditional generation on the ImageNet 256x256 dataset using the LDM-4 model, our W4A4 and W4A3 quantization schemes improve the Inception Score by 12.8% and 467.73%, respectively, over the existing state-of-the-art method.