Characterizing Narrative Content in Web-scale LLM Pretraining Data
For researchers studying LLM pretraining data composition and narrative reasoning, this work provides foundational tools and insights, though it is primarily descriptive and incremental.
This paper presents the first fine-grained study of narrative features in web-scale LLM pretraining data, introducing NarraBERT for narrative prediction and NarraDolma dataset. Key findings include measurable narrative structure at scale, continuous multidimensional narrative features, and unequal distribution across sources, highlighting gaps in current data curation.
The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find (i) narrative structure is measurable at scale across extremely heterogeneous data, (ii) we uncover a continuous, multidimensional narrative structure underlying web text, and (iii) narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.