ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning
This addresses the need for more effective and general QA methods for LLM developers and users, though it appears incremental as it builds on existing retrieval and reasoning approaches.
The paper tackles the problem of improving question-answering performance with large language models by introducing ARR, a method that analyzes question intent, retrieves information, and reasons step-by-step, resulting in consistent outperformance over baselines across 10 diverse QA tasks.
Large language models (LLMs) have demonstrated impressive capabilities on complex evaluation benchmarks, many of which are formulated as question-answering (QA) tasks. Enhancing the performance of LLMs in QA contexts is becoming increasingly vital for advancing their development and applicability. This paper introduces ARR, an intuitive, effective, and general QA solving method that explicitly incorporates three key steps: analyzing the intent of the question, retrieving relevant information, and reasoning step by step. Notably, this paper is the first to introduce intent analysis in QA, which plays a vital role in ARR. Comprehensive evaluations across 10 diverse QA tasks demonstrate that ARR consistently outperforms the baseline methods. Ablation and case studies further validate the positive contributions of each ARR component. Furthermore, experiments involving variations in prompt design indicate that ARR maintains its effectiveness regardless of the specific prompt formulation. Additionally, extensive evaluations across various model sizes, LLM series, and generation settings solidify the effectiveness, robustness, and generalizability of ARR.