Forgetting is Not Erasure: Recovering Latent Knowledge via Transport Keys
For continual learning researchers, this work challenges the prevailing view of forgetting as representational loss and suggests a new direction focused on interface alignment.
The authors show that catastrophic forgetting in continual learning is largely due to interface drift between internal stages rather than permanent erasure, and that compact transport keys can recover most of the original Task A performance after training on Task B (e.g., on split CIFAR-100 with ResNet).
Catastrophic forgetting is often framed as a representational problem: after sequential training, a model appears to lose the features that supported performance on earlier tasks. We challenge the stronger form of this view. Across controlled continual-learning settings, we find that a significant portion of apparent forgetting can be attributed to interface drift between internal stages rather than permanent erasure of task-relevant computation. We study this phenomenon through a stitched evaluation protocol that combines early computation from a post-update network with late computation from its predecessor, optionally mediated by a compact, task-specific transport key. We describe transport keys at a systems level as compact interface-alignment operators estimated from a small set of paired anchor activations and evaluated through model stitching. On split CIFAR-100 with a ResNet-style network, transport keys recover most of the original Task A performance after sequential training on Task B. On a compact vision transformer, we observe a similar recovery pattern. These results suggest that continual learning may require better mechanisms for indexing and re-accessing latent computations, not only methods that prevent weight change.