Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability
This work provides a new feature-centric vocabulary for continual learning, addressing a foundational problem in AI, but it is incremental as it builds on existing mechanistic interpretability concepts.
The paper tackles catastrophic forgetting in continual learning by introducing a mechanistic framework that interprets forgetting as transformations to individual feature encodings, leading to reduced capacity and disrupted readout, and demonstrates this through analysis and experiments including a Vision Transformer on sequential CIFAR-10.
Catastrophic forgetting in continual learning is often measured at the performance or last-layer representation level, overlooking the underlying mechanisms. We introduce a mechanistic framework that offers a geometric interpretation of catastrophic forgetting as the result of transformations to the encoding of individual features. These transformations can lead to forgetting by reducing the allocated capacity of features (worse representation) and disrupting their readout by downstream computations. Analysis of a tractable model formalizes this view, allowing us to identify best- and worst-case scenarios. Through experiments on this model, we empirically test our formal analysis and highlight the detrimental effect of depth. Finally, we demonstrate how our framework can be used in the analysis of practical models through the use of Crosscoders. We present a case study of a Vision Transformer trained on sequential CIFAR-10. Our work provides a new, feature-centric vocabulary for continual learning.