Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
This addresses the challenge of efficient hardware deployment for quantized LLMs by eliminating the need for mixed-precision handling of outliers, though it is incremental as it builds on existing quantization methods.
The paper tackles the problem of sensitive outlier weights in LLM quantization by proposing Noise Perturbation Fine-tuning (NPFT), which reduces their impact through fine-tuning with random perturbations, achieving stable performance improvements and enabling simpler quantizers like RTN to match GPTQ on LLaMA2-7B-4bits.
Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose Noise Perturbation Fine-tuning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.