NarraBench: A Comprehensive Framework for Narrative Benchmarking
This addresses the need for better benchmarks for NLP researchers working on narrative understanding, though it is incremental as it builds on existing work.
The authors tackled the problem of evaluating narrative understanding in NLP by introducing NarraBench, a taxonomy and survey of 78 existing benchmarks, finding that only 27% of narrative tasks are well captured and identifying gaps in areas like narrative events and style.
We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need for new evaluations covering aspects of narrative understanding that are either overlooked in current work or are poorly aligned with existing metrics. Specifically, we estimate that only 27% of narrative tasks are well captured by existing benchmarks, and we note that some areas -- including narrative events, style, perspective, and revelation -- are nearly absent from current evaluations. We also note the need for increased development of benchmarks capable of assessing constitutively subjective and perspectival aspects of narrative, that is, aspects for which there is generally no single correct answer. Our taxonomy, survey, and methodology are of value to NLP researchers seeking to test LLM narrative understanding.