SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
This work addresses the efficiency challenge of deploying LLMs on resource-constrained devices by improving quantization accuracy, though it is incremental as it builds on existing post-training quantization methods.
The paper tackles the problem of activation outliers in 4-bit quantization for Large Language Models (LLMs) by introducing SmoothRot, a post-training quantization technique that combines channel-wise scaling with Hadamard transformations, resulting in a 10-30% reduction in the performance gap between quantized and FP16 models across various tasks.
We present SmoothRot, a novel post-training quantization technique to enhance the efficiency of 4-bit quantization in Large Language Models (LLMs). SmoothRot addresses the critical challenge of massive activation outliers, by integrating channel-wise scaling with Hadamard transformations. Our technique effectively transforms extreme outliers into quantization-friendly activations, significantly improving quantization accuracy. Experiments conducted on popular LLMs (LLaMA2 7B, LLaMA3.1 8B, and Mistral 7B) demonstrate that SmoothRot consistently reduces the performance gap between quantized and FP16 models by approximately 10-30\% across language generation and zero-shot reasoning tasks, without introducing additional inference latency. Code is available at https://github.com/czakop/smoothrot.