IAG: Induction-Augmented Generation Framework for Answering Reasoning Questions
This addresses limitations in retrieval-based approaches for open-domain QA by enhancing reasoning capabilities, though it is incremental as it builds on existing RAG architectures.
The paper tackles the problem of answering implicit reasoning questions in open-domain QA by proposing an Induction-Augmented Generation (IAG) framework that combines retrieved documents with inductive knowledge derived from LLMs, resulting in models that outperform RAG baselines and ChatGPT and achieve first place on CSQA2.0 and StrategyQA leaderboards.
Retrieval-Augmented Generation (RAG), by incorporating external knowledge with parametric memory of language models, has become the state-of-the-art architecture for open-domain QA tasks. However, common knowledge bases are inherently constrained by limited coverage and noisy information, making retrieval-based approaches inadequate to answer implicit reasoning questions. In this paper, we propose an Induction-Augmented Generation (IAG) framework that utilizes inductive knowledge along with the retrieved documents for implicit reasoning. We leverage large language models (LLMs) for deriving such knowledge via a novel prompting method based on inductive reasoning patterns. On top of this, we implement two versions of IAG named IAG-GPT and IAG-Student, respectively. IAG-GPT directly utilizes the knowledge generated by GPT-3 for answer prediction, while IAG-Student gets rid of dependencies on GPT service at inference time by incorporating a student inductor model. The inductor is firstly trained via knowledge distillation and further optimized by back-propagating the generator feedback via differentiable beam scores. Experimental results show that IAG outperforms RAG baselines as well as ChatGPT on two Open-Domain QA tasks. Notably, our best models have won the first place in the official leaderboards of CSQA2.0 (since Nov 1, 2022) and StrategyQA (since Jan 8, 2023).