ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
This work addresses the computational bottleneck in fine-tuning large language models for researchers and practitioners, offering an incremental improvement over existing LoRA variants.
The paper tackles the limitations of low-rank adaptation (LoRA) for fine-tuning large language models, which can hinder effectiveness and slow convergence, by proposing ScaLoRA, a method that accumulates high-rank weight updates from low-rank increments with optimal scaling, achieving consistent performance gains and fast convergence on tasks like natural language understanding and mathematical problem solving.
As large language models (LLMs) continue to scale in size, the computational overhead has become a major bottleneck for task-specific fine-tuning. While low-rank adaptation (LoRA) effectively curtails this cost by confining the weight updates to a low-dimensional subspace, such a restriction can hinder effectiveness and slow convergence. This contribution deals with these limitations by accumulating progressively a high-rank weight update from consecutive low-rank increments. Specifically, the per update optimal low-rank matrix is identified to minimize the loss function and closely approximate full fine-tuning. To endow efficient and seamless optimization without restarting, this optimal choice is formed by appropriately scaling the columns of the original low-rank matrix. Rigorous performance guarantees reveal that the optimal scaling can be found analytically. Extensive numerical tests with popular LLMs scaling up to 12 billion parameters demonstrate a consistent performance gain and fast convergence relative to state-of-the-art LoRA variants on diverse tasks including natural language understanding, commonsense reasoning, and mathematical problem solving.