ChapterBreak: A Challenge Dataset for Long-Range Language Models
This addresses the problem of evaluating long-range language models for researchers, though it is incremental as it focuses on dataset creation rather than model innovation.
The authors tackled the lack of meaningful evaluation for long-range language models by introducing ChapterBreak, a challenge dataset for discourse-level understanding, and found that existing models underperform significantly compared to a task-specific segment-level model.
While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed. To this end, we introduce ChapterBreak, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary and asks it to distinguish the beginning of the ground-truth next chapter from a set of negative segments sampled from the same narrative. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions (e.g., parallel narratives, cliffhanger endings) that require processing global context to comprehend. Experiments on ChapterBreak show that existing LRLMs fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task. We publicly release our ChapterBreak dataset to spur more principled future research into LRLMs.