Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models
This addresses the challenge of robust relational reasoning for users of LLMs in tasks like kinship or spatial reasoning, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of complex relational reasoning in large language models (LLMs) by introducing the Path-of-Thoughts (PoT) framework, which decomposes tasks into graph extraction, path identification, and reasoning stages, resulting in a maximum 21.3% improvement over state-of-the-art baselines on benchmark datasets without fine-tuning or extensive LLM calls.
Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework designed to tackle relation reasoning by decomposing the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a task-agnostic graph that identifies crucial entities, relations, and attributes within the problem context. Subsequently, PoT identifies relevant reasoning chains within the graph corresponding to the posed question, facilitating inference of potential answers. Experimental evaluations on four benchmark datasets, demanding long reasoning chains, demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (maximum 21.3%) without necessitating fine-tuning or extensive LLM calls. Furthermore, as opposed to prior neuro-symbolic methods, PoT exhibits improved resilience against LLM errors by leveraging the compositional nature of graphs.