Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs
This addresses the challenge for smaller LLMs in complex QA scenarios, offering a more efficient alternative to manual annotation and knowledge distillation, though it is incremental as it builds on existing retrieval-augmented methods.
The paper tackles the problem of improving small LLMs' ability to decompose complex questions for retrieval-augmented QA by introducing a framework that uses planning data from knowledge graphs, resulting in enhanced planning capabilities and better performance on complex QA tasks as shown in evaluations on multiple datasets.
Improving the performance of large language models (LLMs) in complex question-answering (QA) scenarios has always been a research focal point. Recent studies have attempted to enhance LLMs' performance by combining step-wise planning with external retrieval. While effective for advanced models like GPT-3.5, smaller LLMs face challenges in decomposing complex questions, necessitating supervised fine-tuning. Previous work has relied on manual annotation and knowledge distillation from teacher LLMs, which are time-consuming and not accurate enough. In this paper, we introduce a novel framework for enhancing LLMs' planning capabilities by using planning data derived from knowledge graphs (KGs). LLMs fine-tuned with this data have improved planning capabilities, better equipping them to handle complex QA tasks that involve retrieval. Evaluations on multiple datasets, including our newly proposed benchmark, highlight the effectiveness of our framework and the benefits of KG-derived planning data.