DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
This addresses a critical bottleneck in deploying efficient, low-bit LLMs for practical applications, though it is an incremental improvement over existing outlier mitigation techniques.
The paper tackles the problem of outlier activations in quantized large language models (LLMs), which degrade performance, by introducing DuQuant, a method using rotation and permutation transformations to mitigate both massive and normal outliers, achieving state-of-the-art results with 4-bit weight-activation quantization across various LLMs and tasks.
Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.