Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
For researchers in continual learning, this work provides a principled way to mitigate forgetting in low-rank adaptations, though it is incremental over existing methods.
The paper identifies that low-rank adaptations in continual learning suffer from imbalanced singular value spectra, which cause forgetting. They propose a method to balance components via constrained optimization on a Stiefel manifold, outperforming baselines.
Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.