Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models
This reveals the complexity of knowledge storage in language models, which is important for researchers working on model interpretability and editing, though it is incremental as it builds on prior localization studies.
The study investigated differences between entity and relational knowledge in language models through knowledge editing, finding they cannot be directly transferred, and used causal analysis to show relational knowledge is significantly encoded in attention modules, not just MLP weights.
Large language models encapsulate knowledge and have demonstrated superior performance on various natural language processing tasks. Recent studies have localized this knowledge to specific model parameters, such as the MLP weights in intermediate layers. This study investigates the differences between entity and relational knowledge through knowledge editing. Our findings reveal that entity and relational knowledge cannot be directly transferred or mapped to each other. This result is unexpected, as logically, modifying the entity or the relation within the same knowledge triplet should yield equivalent outcomes. To further elucidate the differences between entity and relational knowledge, we employ causal analysis to investigate how relational knowledge is stored in pre-trained models. Contrary to prior research suggesting that knowledge is stored in MLP weights, our experiments demonstrate that relational knowledge is also significantly encoded in attention modules. This insight highlights the multifaceted nature of knowledge storage in language models, underscoring the complexity of manipulating specific types of knowledge within these models.