FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models
This work addresses the problem of evaluating complex reasoning capabilities in large language models for researchers and developers, but it is incremental as it builds on existing benchmarking efforts.
The authors tackled the lack of resources for evaluating complex multi-hop, multi-document reasoning in large language models by introducing FanOutQA, a benchmark dataset based on English Wikipedia, and found that contemporary models like GPT-4 and LLaMA 2 still struggle with reasoning over inter-document dependencies in long contexts.
One type of question that is commonly found in day-to-day scenarios is ``fan-out'' questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over inter-document dependencies in a long context. We provide our dataset and open-source tools to run models to encourage evaluation at https://fanoutqa.com