Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation
This work addresses the challenge of efficient and robust domain adaptation for pre-trained models like CLIP and DINO-ViT, offering an incremental improvement over existing parameter-efficient fine-tuning methods.
The paper tackles the problem of catastrophic forgetting and reduced robustness in fine-tuning deep learning models by proposing a parameter-efficient method that selectively activates Low-Rank Adaptation blocks, achieving effective adaptation with as few as 5% active blocks while maintaining performance and knowledge retention.
Adapting deep learning models to new domains often requires computationally intensive retraining and risks catastrophic forgetting. While fine-tuning enables domain-specific adaptation, it can reduce robustness to distribution shifts, impacting out-of-distribution (OOD) performance. Pre-trained zero-shot models like CLIP offer strong generalization but may suffer degraded robustness after fine-tuning. Building on Task Adaptive Parameter Sharing (TAPS), we propose a simple yet effective extension as a parameter-efficient fine-tuning (PEFT) method, using an indicator function to selectively activate Low-Rank Adaptation (LoRA) blocks. Our approach minimizes knowledge loss, retains its generalization strengths under domain shifts, and significantly reduces computational costs compared to traditional fine-tuning. We demonstrate that effective fine-tuning can be achieved with as few as 5\% of active blocks, substantially improving efficiency. Evaluations on pre-trained models such as CLIP and DINO-ViT demonstrate our method's broad applicability and effectiveness in maintaining performance and knowledge retention.