Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
This work improves the convergence speed and performance of LoRA fine-tuning for large language models, which is a significant incremental gain for practitioners using this widely adopted method.
This paper addresses the overparameterization in Low-Rank Adaptation (LoRA) by showing that different low-rank factor pairs, while yielding the same adapted weight matrix, have significantly different condition numbers. They introduce Balanced Low-Rank Adaptation (BaLoRA), which projects iterates onto a balanced manifold to improve the conditioning of the loss landscape, leading to faster convergence and superior performance compared to standard LoRA.
Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.