HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs
This addresses the challenge of enabling LLMs to answer complex questions more efficiently, representing an incremental improvement over existing structured knowledge integration methods.
The paper tackles the problem of LLMs struggling with complex multi-hop questions by proposing a context-aware, query-relevant knowledge graph to reduce token usage by up to 67% compared to state-of-the-art methods, leading to consistent improvements across metrics on benchmark datasets like HotpotQA and MuSiQue.
Given unstructured text, Large Language Models (LLMs) are adept at answering simple (single-hop) questions. However, as the complexity of the questions increase, the performance of LLMs degrade. We believe this is due to the overhead associated with understanding the complex question followed by filtering and aggregating unstructured information in the raw text. Recent methods try to reduce this burden by integrating structured knowledge triples into the raw text, aiming to provide a structured overview that simplifies information processing. However, this simplistic approach is query-agnostic and the extracted facts are ambiguous as they lack context. To address these drawbacks and to enable LLMs to answer complex (multi-hop) questions with ease, we propose to use a knowledge graph (KG) that is context-aware and is distilled to contain query-relevant information. The use of our compressed distilled KG as input to the LLM results in our method utilizing up to $67\%$ fewer tokens to represent the query relevant information present in the supporting documents, compared to the state-of-the-art (SoTA) method. Our experiments show consistent improvements over the SoTA across several metrics (EM, F1, BERTScore, and Human Eval) on two popular benchmark datasets (HotpotQA and MuSiQue).