HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations
This work addresses deployment challenges on resource-constrained devices for diffusion models, representing an incremental improvement in quantization techniques.
The paper tackled the problem of high memory and computational demands in diffusion models for image generation by proposing HadaNorm, a linear transformation method that reduces quantization error and outperforms state-of-the-art methods in enabling aggressive activation quantization.
Diffusion models represent the cutting edge in image generation, but their high memory and computational demands hinder deployment on resource-constrained devices. Post-Training Quantization (PTQ) offers a promising solution by reducing the bitwidth of matrix operations. However, standard PTQ methods struggle with outliers, and achieving higher compression often requires transforming model weights and activations before quantization. In this work, we propose HadaNorm, a novel linear transformation that extends existing approaches by both normalizing channels activations and applying Hadamard transforms to effectively mitigate outliers and enable aggressive activation quantization. We demonstrate that HadaNorm consistently reduces quantization error across the various components of transformer blocks, outperforming state-of-the-art methods.