LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)
This provides theoretical insights into why LoRA fine-tuning usually succeeds, addressing a gap for researchers and practitioners using large foundation models.
The paper tackles the limited theoretical understanding of LoRA training dynamics by analyzing its loss landscape without restrictive assumptions, showing that in realistic setups, it converges to a low-rank global minimizer or a distinct high-rank solution, with zero-initialization and weight decay inducing an implicit bias toward the low-rank region.
Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a "special regime", which includes idealized setups where linearization arguments hold, and a "generic regime" representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space -- where global minima lie -- thus shedding light on why LoRA training usually succeeds in finding global minima.