On the Limitations of Rank-One Model Editing in Answering Multi-hop Questions
This addresses a critical bottleneck in knowledge editing for multi-hop question answering, though it's an incremental improvement over existing methods.
The paper identifies limitations of Rank-One Model Editing (ROME) in handling multi-hop reasoning tasks, showing it struggles with three failure modes including poor generalization and overfitting. The proposed Redundant Editing strategy improves accuracy on 2-hop questions by at least 15.5 percentage points (a 96% increase over single-edit approaches).
Recent advances in Knowledge Editing (KE), particularly Rank-One Model Editing (ROME), show superior efficiency over fine-tuning and in-context learning for updating single-hop facts in transformers. However, these methods face significant challenges when applied to multi-hop reasoning tasks requiring knowledge chaining. In this work, we study the effect of editing knowledge with ROME on different layer depths and identify three key failure modes. First, the "hopping-too-late" problem occurs as later layers lack access to necessary intermediate representations. Second, generalization ability deteriorates sharply when editing later layers. Third, the model overfits to edited knowledge, incorrectly prioritizing edited-hop answers regardless of context. To mitigate the issues of "hopping-too-late" and generalisation decay, we propose Redundant Editing, a simple yet effective strategy that enhances multi-hop reasoning. Our experiments demonstrate that this approach can improve accuracy on 2-hop questions by at least 15.5 percentage points, representing a 96% increase over the previous single-edit strategy, while trading off some specificity and language naturalness.