Energy-Structured Low-Rank Adaptation for Continual Learning
For continual learning practitioners, this method improves knowledge compaction and capacity utilization, outperforming existing orthogonal subspace approaches.
The paper addresses catastrophic forgetting in continual learning by proposing E²-LoRA, a method that concentrates knowledge into leading low-rank subspaces to free capacity for future tasks. It achieves state-of-the-art performance across multiple benchmarks.
While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose \textbf{E}nergy-Concentrated and \textbf{E}nergy-Ordered \textbf{Lo}w-\textbf{R}ank \textbf{A}daptation (E$^2$-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E$^2$-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E$^2$-LoRA achieves state-of-the-art performance.