On the Proper Treatment of Units in Surprisal Theory
For psycholinguists and NLP researchers, this work clarifies a methodological confusion in surprisal-based analyses, though it is primarily a conceptual clarification rather than a novel empirical finding.
Surprisal theory links processing effort to predictability, but empirical work often underspecifies the unit of analysis. This paper disentangles unit definition from region-of-interest choices, providing a unified framework that treats tokenization as an implementation detail.
Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.