One Thousand and One Pairs: A "novel" challenge for long-context language models
This addresses the need for more realistic benchmarks to assess long-context LLMs' reasoning capabilities beyond surface-level retrieval, though it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating long-context language models' ability to retrieve, synthesize, and reason over book-length inputs by creating NoCha, a dataset of 1,001 true/false claim pairs about fictional books, and found that while humans excel, all ten tested LLMs struggle, with GPT-4o achieving only 55.8% accuracy and no open-weight model performing above random chance.
Synthetic long-context LLM benchmarks (e.g., "needle-in-the-haystack") test only surface-level retrieval capabilities, but how well can long-context LLMs retrieve, synthesize, and reason over information across book-length inputs? We address this question by creating NoCha, a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books, written by human readers of those books. In contrast to existing long-context benchmarks, our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. Our experiments show that while human readers easily perform this task, it is enormously challenging for all ten long-context LLMs that we evaluate: no open-weight model performs above random chance (despite their strong performance on synthetic benchmarks), while GPT-4o achieves the highest accuracy at 55.8%. Further analysis reveals that (1) on average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning; (2) model-generated explanations for their decisions are often inaccurate even for correctly-labeled claims; and (3) models perform substantially worse on speculative fiction books that contain extensive world-building. The methodology proposed in NoCha allows for the evolution of the benchmark dataset and the easy analysis of future models.