PLAN: Proactive Low-Rank Allocation for Continual Learning
This addresses the problem of catastrophic forgetting for continual learning with foundation models, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of catastrophic forgetting in continual learning by proposing PLAN, a framework that proactively manages task-specific subspaces using orthogonal basis vectors and a perturbation-based optimization strategy. Empirical results show PLAN consistently outperforms existing methods and establishes new state-of-the-art performance on standard continual learning benchmarks.
Continual learning (CL) requires models to continuously adapt to new tasks without forgetting past knowledge. In this work, we propose \underline{P}roactive \underline{L}ow-rank \underline{A}llocatio\underline{N} (PLAN), a framework that extends Low-Rank Adaptation (LoRA) to enable efficient and interference-aware fine-tuning of large pre-trained models in CL settings. PLAN proactively manages the allocation of task-specific subspaces by introducing orthogonal basis vectors for each task and optimizing them through a perturbation-based strategy that minimizes conflicts with previously learned parameters. Furthermore, PLAN incorporates a novel selection mechanism that identifies and assigns basis vectors with minimal sensitivity to interference, reducing the risk of degrading past knowledge while maintaining efficient adaptation to new tasks. Empirical results on standard CL benchmarks demonstrate that PLAN consistently outperforms existing methods, establishing a new state-of-the-art for continual learning with foundation models.