Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysis
This addresses the challenge of lightweight knowledge updates for compositional reasoning tasks in LLMs, representing a novel method for a known bottleneck.
The paper tackles the problem of updating outdated information in Large Language Models for multi-hop question answering by proposing a knowledge editor called CHECK, which uses semantic analysis to revise reasoning chains, achieving an average 22.8% improved accuracy on four datasets.
Large Language Models (LLMs) require lightweight avenues of updating stored information that has fallen out of date. Knowledge Editing (KE) approaches have been successful in updating model knowledge for simple factual queries but struggle with handling tasks that require compositional reasoning such as multi-hop question answering (MQA). We observe that existing knowledge editors leverage decompositional techniques that result in illogical reasoning processes. In this paper, we propose a knowledge editor for MQA based on semantic analysis called CHECK. Our framework is based on insights from an analogy between compilers and reasoning using LLMs. Similar to how source code is first compiled before being executed, we propose to semantically analyze reasoning chains before executing the chains to answer questions. Reasoning chains with semantic errors are revised to ensure consistency through logic optimization and re-prompting the LLM model at a higher temperature. We evaluate the effectiveness of CHECK against five state-of-the-art frameworks on four datasets and achieve an average 22.8% improved MQA accuracy.