Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
This addresses the problem of efficient fine-tuning for large models, offering a more memory-efficient alternative to existing LoRA methods, though it appears incremental in nature.
The paper tackles the gap between full-step training with low-rank projections and LoRA fine-tuning by proposing LoRSum, a memory-efficient method that casts LoRA optimization as a proximal sub-problem solved with alternating least squares updates. Experiments on tasks like CIFAR-100, GLUE, and WikiText-103 show the method matches or improves LoRA baselines with modest compute overhead while avoiding full-matrix SVD projections.
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning. We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent by casting LoRA optimization as a proximal sub-problem and solving it efficiently with alternating least squares updates, which we prove to be an implicit block power method. We recover several recently proposed preconditioning methods for LoRA as special cases, and show that LoRSum can also be used for updating a low-rank momentum. In order to address full steps with preconditioned gradient descent, we propose a scaled variant of LoRSum that uses structured metrics such as K-FAC and Shampoo, and we show that storing the diagonal of these metrics still allows them to perform well while remaining memory-efficient. Experiments on a synthetic task, CIFAR-100, and language-model fine-tuning on GLUE, SQuAD v2, and WikiText-103, show that our method can match or improve LoRA baselines given modest compute overhead, while avoiding full-matrix SVD projections and retaining LoRA-style parameter efficiency.