GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
This work addresses the need for efficient fine-tuning of LLMs, offering practical benefits for deployment, though it is incremental as it builds on existing PEFT and sensitivity analysis concepts.
The paper tackles the problem of parameter-efficient fine-tuning for large language models by introducing GIFT-SW, a method that updates only salient weights while injecting Gaussian noise into others, resulting in outperforming full fine-tuning and other PEFT methods under the same computational budget and recovering performance in quantized models.
Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developeda generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.