Curvature-Guided LoRA: Steering in the pretrained NTK subspace
This addresses the performance gap in efficient adaptation of large pretrained models for NLP tasks, though it appears incremental as it builds on existing LoRA variants.
The paper tackled the problem of parameter-efficient fine-tuning (PEFT) methods like LoRA underperforming compared to full fine-tuning by introducing a prediction alignment objective to match outputs, resulting in Curvature-Guided LoRA (CG-LoRA) that improved performance and convergence speed on natural language understanding benchmarks.
Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models but often fall short of full fine-tuning performance. Existing approaches focus on aligning parameter updates, which only indirectly control model predictions. In this work, we introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs. We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient. Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information. Our method is computationally efficient and avoids explicit second-order matrix construction. Preliminary experiments on standard natural language understanding benchmarks demonstrate improved performance and faster convergence compared to existing LoRA variants.