QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
This enables more efficient deployment of LLMs for applications requiring low memory and computational resources, though it is incremental as it builds on existing quantization techniques.
The paper tackles the problem of quantizing large language models (LLMs) to 4 bits for efficient inference by introducing QuaRot, a quantization scheme based on rotations that removes outliers without changing model output, resulting in a 4-bit LLaMa2-70B model with at most 0.47 WikiText-2 perplexity loss and 99% zero-shot performance retention.
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.