Understanding Empirical Unlearning with Combinatorial Interpretability
This addresses the challenge of verifying unlearning in large models, which is crucial for privacy and safety, but is incremental as it applies an existing interpretability framework to a specific unlearning context.
The paper tackled the problem of understanding whether unlearning methods truly remove knowledge from pretrained models or merely hide it, by using combinatorial interpretability to inspect model weights in a two-layer neural network setting, finding that knowledge often persists and can be recovered through fine-tuning.
While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large foundation models are far from interpretable, understanding whether and how such knowledge persists remains a significant challenge. To address this, we turn to the recently developed framework of combinatorial interpretability. This framework, designed for two-layer neural networks, enables direct inspection of the knowledge encoded in the model weights. We reproduce baseline unlearning methods within the combinatorial interpretability setting and examine their behavior along two dimensions: (i) whether they truly remove knowledge of a target concept (the concept we wish to remove) or merely inhibit its expression while retaining the underlying information, and (ii) how easily the supposedly erased knowledge can be recovered through various fine-tuning operations. Our results shed light within a fully interpretable setting on how knowledge can persist despite unlearning and when it might resurface.