NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
This provides a diagnostic tool for researchers to test multi-hop reasoning failures in long contexts, though it is incremental as it builds on existing benchmark concepts.
The authors tackled the problem of large language models struggling with multi-hop reasoning in long narrative contexts by introducing NovelHopQA, a benchmark evaluating 1-4 hop question answering over 64k-128k-token excerpts from 83 novels, and found consistent accuracy drops with increased hops and context length even for state-of-the-art models.
Current large language models (LLMs) struggle to answer questions that span tens of thousands of tokens, especially when multi-hop reasoning is involved. While prior benchmarks explore long-context comprehension or multi-hop reasoning in isolation, none jointly vary context length and reasoning depth in natural narrative settings. We introduce NovelHopQA, the first benchmark to evaluate 1-4 hop QA over 64k-128k-token excerpts from 83 full-length public-domain novels. A keyword-guided pipeline builds hop-separated chains grounded in coherent storylines. We evaluate seven state-of-the-art models and apply oracle-context filtering to ensure all questions are genuinely answerable. Human annotators validate both alignment and hop depth. We additionally present retrieval-augmented generation (RAG) evaluations to test model performance when only selected passages are provided instead of the full context. We noticed consistent accuracy drops with increased hops and context length increase, even for frontier models-revealing that sheer scale does not guarantee robust reasoning. Failure-mode analysis highlights common breakdowns such as missed final-hop integration and long-range drift. NovelHopQA offers a controlled diagnostic setting to test multi-hop reasoning at scale. All code and datasets are available at https://novelhopqa.github.io.