Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
For continual learning researchers, this work offers a principled understanding of forgetting in the NTK regime and a targeted regularizer, though it is incremental over existing NTK-based analyses.
The paper provides a function-space theory for catastrophic forgetting in continual adaptation, showing that forgetting is low-rank and concentrated in a few eigenmodes of the old-task NTK. It derives a closed-form predictor for forgetting and proposes a spectral regularizer that outperforms baselines.
Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision; for nonlinear adapters/full fine-tuning, it is a local NTK approximation. The same expression reveals that forgetting concentrates in a small number of old-task NTK eigenmodes and under frozen linear heads gives a Kronecker scaling rule for the vulnerable rank. These results clarify the relation to prior NTK-overlap theory, explain why parameter-space regularizers can miss output-space interference, and motivate a targeted spectral regularizer.