PropMEND: Hypernetworks for Knowledge Propagation in LLMs
This addresses the limitation of knowledge editing in LLMs for downstream reasoning tasks, though it shows reduced performance on unseen relations, indicating incremental progress.
The paper tackles the problem of knowledge propagation in large language models, where existing knowledge editing techniques fail to enable reasoning with injected knowledge. The authors present PropMEND, a hypernetwork-based approach that improves accuracy on multi-hop questions by almost 2x on the RippleEdit dataset.
Knowledge editing techniques for large language models (LLMs) can inject knowledge that is later reproducible verbatim, but they fall short on propagating that knowledge: models cannot answer questions that require reasoning with the injected knowledge. We present a hypernetwork-based approach for knowledge propagation, named PropMEND, where we meta-learn how to modify gradients of a language modeling loss to encourage injected information to propagate. Our approach extends the meta-objective of MEND [29] so that gradient updates on knowledge are transformed to enable answering multi-hop questions involving that knowledge. We show improved performance on the RippleEdit dataset, showing almost 2x accuracy on challenging multi-hop questions whose answers are not explicitly stated in the injected fact. We further introduce a new dataset, Controlled RippleEdit, to evaluate the generalization of our hypernetwork, testing knowledge propagation along relations and entities unseen during hypernetwork training. PropMEND still outperforms existing approaches in unseen entity-relation pairs, yet the performance gap decreases substantially, suggesting future work in propagating knowledge to a wide range of relations.