Turning LLM Activations Quantization-Friendly
This work addresses the problem of reducing serving costs for LLMs through better quantization, but it appears incremental as it builds on existing techniques like smoothing and rotation.
The paper tackles the challenge of quantizing LLM activations due to outliers by analyzing their impact on quantization error and proposing a hybrid method combining channel-wise scaling and rotation, achieving improved quantization performance.
Quantization effectively reduces the serving costs of Large Language Models (LLMs) by speeding up data movement through compressed parameters and enabling faster operations via integer arithmetic. However, activating integer arithmetic requires quantizing both weights and activations, which poses challenges due to the significant outliers in LLMs that increase quantization error. In this work, we investigate these outliers with an emphasis on their effect on layer-wise quantization error, then examine how smoothing and rotation transform the observed values. Our primary contributions include introducing a new metric to measure and visualize quantization difficulty based on channel magnitudes, as well as proposing a hybrid approach that applies channel-wise scaling before rotation, supported by a mathematical formulation of its benefits.