Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editing
This addresses a critical challenge for researchers and practitioners in AI who rely on knowledge editing to update models, revealing that current methods are superficial and may not effectively modify underlying knowledge.
The paper tackles the problem of knowledge editing in language models being deceptive, where models edited by existing algorithms still generate original knowledge despite high conventional metric scores, and identifies key factors like residual streams and specific attention modules that cause this issue.
Knowledge editing, which aims to update the knowledge encoded in language models, can be deceptive. Despite the fact that many existing knowledge editing algorithms achieve near-perfect performance on conventional metrics, the models edited by them are still prone to generating original knowledge. This paper introduces the concept of "superficial editing" to describe this phenomenon. Our comprehensive evaluation reveals that this issue presents a significant challenge to existing algorithms. Through systematic investigation, we identify and validate two key factors contributing to this issue: (1) the residual stream at the last subject position in earlier layers and (2) specific attention modules in later layers. Notably, certain attention heads in later layers, along with specific left singular vectors in their output matrices, encapsulate the original knowledge and exhibit a causal relationship with superficial editing. Furthermore, we extend our analysis to the task of superficial unlearning, where we observe consistent patterns in the behavior of specific attention heads and their corresponding left singular vectors, thereby demonstrating the robustness and broader applicability of our methodology and conclusions. Our code is available here.