Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
This work addresses a fundamental problem in adapting large pre-trained models for practitioners needing efficient fine-tuning with minimal forgetting, though it is incremental as it builds on existing LoRA methods.
The paper tackles the challenge of balancing task-specific performance gains against catastrophic forgetting in Low-Rank Adaptation (LoRA) methods by analyzing performance-forgetting trade-offs using principal components as initialization. It finds that fine-tuning intermediate components leads to better balance and robustness, and demonstrates improved accuracy and reduced forgetting in computer vision and NLP tasks.
Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational and memory constraints. However, they face a fundamental challenge in balancing task-specific performance gains against catastrophic forgetting of pre-trained knowledge, where existing methods provide inconsistent recommendations. This paper presents a comprehensive analysis of the performance-forgetting trade-offs inherent in low-rank adaptation using principal components as initialization. Our investigation reveals that fine-tuning intermediate components leads to better balance and show more robustness to high learning rates than first (PiSSA) and last (MiLoRA) components in existing work. Building on these findings, we provide a practical approach for initialization of LoRA that offers superior trade-offs. We demonstrate in a thorough empirical study on a variety of computer vision and NLP tasks that our approach improves accuracy and reduces forgetting, also in continual learning scenarios.