Evaluation of retrieval-based QA on QUEST-LOFT
This addresses a key limitation in grounded QA for AI systems, though it appears incremental as it builds on existing RAG methods.
The paper tackled the problem of retrieval-augmented generation (RAG) struggling with questions requiring information from many documents or complex reasoning, as shown by poor performance on the QUEST-LOFT benchmark, and demonstrated that optimizing RAG with structured output and re-verification significantly outperforms long-context approaches.
Despite the popularity of retrieval-augmented generation (RAG) as a solution for grounded QA in both academia and industry, current RAG methods struggle with questions where the necessary information is distributed across many documents or where retrieval needs to be combined with complex reasoning. Recently, the LOFT study has shown that this limitation also applies to approaches based on long-context language models, with the QUEST benchmark exhibiting particularly large headroom. In this paper, we provide an in-depth analysis of the factors contributing to the poor performance on QUEST-LOFT, publish updated numbers based on a thorough human evaluation, and demonstrate that RAG can be optimized to significantly outperform long-context approaches when combined with a structured output format containing reasoning and evidence, optionally followed by answer re-verification.