Causal Reasoning in Large Language Models: A Knowledge Graph Approach
This work addresses the challenge of optimizing reasoning capabilities in LLMs for AI researchers, though it appears incremental as it builds on existing knowledge graph and prompting techniques.
The paper tackled the problem of improving large language model performance by comparing retrieval and reasoning strategies, proposing a knowledge graph-based random-walk method that enhanced reasoning and performance on commonsense question answering, with findings showing that incorporating irrelevant sentences via this method boosted results.
Large language models (LLMs) typically improve performance by either retrieving semantically similar information, or enhancing reasoning abilities through structured prompts like chain-of-thought. While both strategies are considered crucial, it remains unclear which has a greater impact on model performance or whether a combination of both is necessary. This paper answers this question by proposing a knowledge graph (KG)-based random-walk reasoning approach that leverages causal relationships. We conduct experiments on the commonsense question answering task that is based on a KG. The KG inherently provides both relevant information, such as related entity keywords, and a reasoning structure through the connections between nodes. Experimental results show that the proposed KG-based random-walk reasoning method improves the reasoning ability and performance of LLMs. Interestingly, incorporating three seemingly irrelevant sentences into the query using KG-based random-walk reasoning enhances LLM performance, contrary to conventional wisdom. These findings suggest that integrating causal structures into prompts can significantly improve reasoning capabilities, providing new insights into the role of causality in optimizing LLM performance.