On the Convergence of Stochastic Low-Rank Adaptation
For practitioners using LoRA for fine-tuning large models, this work provides theoretical guarantees and improved convergence rates for stochastic optimization, though the improvements are incremental over existing analysis.
This paper improves the convergence analysis of LoRA, showing that deterministic LoRA-GD requires O(ε^{-4}) full-gradient evaluations to find an ε-stationary point, and introduces stochastic LoRA methods (LoRA-NSGDM and LoRA-STORM) with oracle complexities of O(ε^{-8}) and O(ε^{-6}) respectively.
Low-rank adaptation (LoRA) optimizes $J(B,A)=\mathcal L(W_\mathrm{base}+sBA)$ over two adapters $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ that form a low-rank update to a frozen pretrained weight matrix $W_\mathrm{base} \in \mathbb{R}^{m \times n}$. The prior analysis shows LoRA-GD takes $\exp\{\mathcal{O}(ε^{-2})\}$ oracle calls to find an $ε$-stationary point such that $\|\nabla J(B,A)\|\leq ε$ in the deterministic setting. We sharpen the analysis and show that $\mathcal{O}(ε^{-4})$ full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an $ε$-stationary point with $\mathcal{O}(ε^{-8})$ stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to $\mathcal{O}(ε^{-6})$.