LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA
This addresses the need for more reliable benchmarks in narrative QA research, though it is incremental as it refines an existing dataset.
The authors tackled the problem of unreliable evaluation in long-document narrative QA by creating LiteraryQA, a high-quality subset of NarrativeQA, and found that LLM-as-a-Judge evaluations strongly agree with human rankings while n-gram-based metrics have low correlation.
Question Answering (QA) on narrative text poses a unique challenge to current systems, requiring a deep understanding of long, complex documents. However, the reliability of NarrativeQA, the most widely used benchmark in this domain, is hindered by noisy documents and flawed QA pairs. In this work, we introduce LiteraryQA, a high-quality subset of NarrativeQA focused on literary works. Using a human- and LLM-validated pipeline, we identify and correct low-quality QA samples while removing extraneous text from source documents. We then carry out a meta-evaluation of automatic metrics to clarify how systems should be evaluated on LiteraryQA. This analysis reveals that all n-gram-based metrics have a low system-level correlation to human judgment, while LLM-as-a-Judge evaluations, even with small open-weight models, can strongly agree with the ranking identified by humans. Finally, we benchmark a set of long-context LLMs on LiteraryQA. We release our code and data at https://github.com/SapienzaNLP/LiteraryQA.