A Step Closer to Comprehensive Answers: Constrained Multi-Stage Question Decomposition with Large Language Models
This addresses the issue of unreliable answers in AI systems for users needing accurate multi-hop reasoning, though it is incremental as it builds on existing methods like ReAct.
The paper tackles the problem of hallucinations in large language models for complex question answering by introducing the Decompose-and-Query framework, which achieved an F1 score of 59.6% on HotPotQA and performed comparably to ChatGPT in 67% of cases on a custom dataset.
While large language models exhibit remarkable performance in the Question Answering task, they are susceptible to hallucinations. Challenges arise when these models grapple with understanding multi-hop relations in complex questions or lack the necessary knowledge for a comprehensive response. To address this issue, we introduce the "Decompose-and-Query" framework (D&Q). This framework guides the model to think and utilize external knowledge similar to ReAct, while also restricting its thinking to reliable information, effectively mitigating the risk of hallucinations. Experiments confirm the effectiveness of D&Q: On our ChitChatQA dataset, D&Q does not lose to ChatGPT in 67% of cases; on the HotPotQA question-only setting, D&Q achieved an F1 score of 59.6%. Our code is available at https://github.com/alkaidpku/DQ-ToolQA.