LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
This addresses the problem of efficient fine-tuning for large pre-trained models, offering a more effective parameter-efficient method that behaves like full fine-tuning, which is incremental but improves upon existing adapter-based approaches.
The paper tackles the performance gap and slower convergence of Low-Rank Adaptation (LoRA) compared to full fine-tuning by introducing LoFT, which aligns optimizer dynamics with full-model updates, resulting in substantial narrowing of the performance gap and consistent outperformance of standard LoRA methods without increasing inference cost.
Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all weights. While LoRA dramatically reduces trainable parameters with little overhead, it can still underperform full fine-tuning in accuracy and often converges more slowly. We introduce LoFT, a novel low-rank adaptation method that behaves like full fine-tuning by aligning the optimizer's internal dynamics with those of updating all model weights. LoFT not only learns weight updates in a low-rank subspace (like LoRA) but also properly projects the optimizer's first and second moments (Adam's momentum and variance) into the same subspace, mirroring full-model updates. By aligning the low-rank update itself with the full update, LoFT eliminates the need for tuning extra hyperparameters, e.g., LoRA scaling factor $α$. Empirically, this approach substantially narrows the performance gap between adapter-based tuning and full fine-tuning and consistently outperforms standard LoRA-style methods, all without increasing inference cost.