A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
This work addresses a bottleneck in parameter-efficient fine-tuning for practitioners using LoRA, offering an incremental improvement to enhance performance with higher ranks.
The paper tackled the problem of slowed learning and stunted performance in Low-Rank Adapters (LoRA) for fine-tuning large language models due to an inappropriate scaling factor, and proposed rank-stabilized LoRA (rsLoRA) by dividing adapters by the square root of the rank, enabling a trade-off where larger ranks improve fine-tuning performance without increasing inference cost.
As large language models (LLMs) have become increasingly compute and memory intensive, parameter-efficient fine-tuning (PEFT) methods are now a common strategy to fine-tune LLMs. A popular PEFT method is Low-Rank Adapters (LoRA), which adds trainable low-rank "adapters" to selected layers. Each adapter consists of a low-rank matrix product, multiplicatively scaled by a rank-dependent factor. This scaling factor, which divides adapters by a factor of the rank, results in slowed learning and stunted performance for LoRA with higher-rank adapters. Consequently, the use of LoRA in practice has generally been limited to very low ranks. In this work, we study the impact of the scaling factor on the learning process and prove that LoRA adapters should be divided by a factor of the square root of the rank. Modifying LoRA with the appropriate scaling factor, which we call the rank-stabilized LoRA (rsLoRA) method, easily provides for a fine-tuning compute/performance trade-off, where larger ranks can be used to trade off increased computational resources during training for better fine-tuning performance, with no change in inference computing cost.