The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
This addresses a methodological gap in cognitive modeling for researchers, though it is incremental as it focuses on a specific factor within existing surprisal frameworks.
The study investigated how token granularity in language models affects surprisal's ability to predict human reading difficulty, finding that a vocabulary size of 8,000 tokens yielded the most predictive surprisal for naturalistic text, while coarser tokens increased sensitivity to garden-path effects.
Word-by-word language model surprisal is often used to model the incremental processing of human readers, which raises questions about how various choices in language modeling influence its predictive power. One factor that has been overlooked in cognitive modeling is the granularity of subword tokens, which explicitly encodes information about word length and frequency, and ultimately influences the quality of vector representations that are learned. This paper presents experiments that manipulate the token granularity and evaluate its impact on the ability of surprisal to account for processing difficulty of naturalistic text and garden-path constructions. Experiments with naturalistic reading times reveal a substantial influence of token granularity on surprisal, with tokens defined by a vocabulary size of 8,000 resulting in surprisal that is most predictive. In contrast, on garden-path constructions, language models trained on coarser-grained tokens generally assigned higher surprisal to critical regions, suggesting a greater sensitivity to garden-path effects than previously reported. Taken together, these results suggest a large role of token granularity on the quality of language model surprisal for cognitive modeling.