LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning
This addresses the challenge of maintaining robust performance in fine-tuning large models for diverse tasks, though it appears incremental as it builds on existing low-rank adaptation methods.
The paper tackled the problem of robust parameter-efficient fine-tuning under domain shifts by proposing the LARGO algorithm, which achieved state-of-the-art performance across in-domain and out-of-distribution scenarios with significantly lower computational overhead.
The advent of parameter-efficient fine-tuning methods has significantly reduced the computational burden of adapting large-scale pretrained models to diverse downstream tasks. However, existing approaches often struggle to achieve robust performance under domain shifts while maintaining computational efficiency. To address this challenge, we propose Low-rAnk Regulated Gradient Projection (LARGO) algorithm that integrates dynamic constraints into low-rank adaptation methods. Specifically, LARGO incorporates parallel trainable gradient projections to dynamically regulate layer-wise updates, retaining the Out-Of-Distribution robustness of pretrained model while preserving inter-layer independence. Additionally, it ensures computational efficiency by mitigating the influence of gradient dependencies across layers during weight updates. Besides, through leveraging singular value decomposition of pretrained weights for structured initialization, we incorporate an SVD-based initialization strategy that minimizing deviation from pretrained knowledge. Through extensive experiments on diverse benchmarks, LARGO achieves state-of-the-art performance across in-domain and out-of-distribution scenarios, demonstrating improved robustness under domain shifts with significantly lower computational overhead compared to existing PEFT methods. The source code will be released soon.