From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation
This work addresses the problem of annotation quality for NLP researchers working on graph-based narrative analysis, but it is incremental as it builds on existing methods for narrative representation and evaluation.
The paper tackles the challenge of annotating and evaluating narratives in news discourse by introducing a narrative graph annotation framework that integrates qualitative content analysis to reduce annotation errors, showing that lenient metrics overestimate reliability and locally-constrained representations reduce variability in a dataset of inflation narratives.
Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a $6\times3$ factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf's $α$), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf's $α$ are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.