Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
This work addresses a bottleneck in parameter-efficient fine-tuning for large language models, particularly in resource-constrained environments, by improving upon LoRA with a more efficient and stable method.
The paper tackled the challenge of inefficient and unstable rank allocation in Low-Rank Adaptation (LoRA) for fine-tuning large language models by proposing Sensitivity-LoRA, which dynamically allocates ranks based on weight sensitivities using second-order derivatives, resulting in robust effectiveness, efficiency, and stability across diverse tasks and benchmarks.
Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks.