TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
For continual learning practitioners, TailLoR offers a novel approach to mitigate catastrophic forgetting while maintaining parameter efficiency.
TailLoR introduces a parameter-efficient continual learning method that applies low-rank updates to the singular value matrix of pre-trained weights, using a soft spectral penalty to protect principal components. This reduces interference and improves performance on continual learning benchmarks.
Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.