Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly
This is an incremental finding that shows a known trend in reading times also applies to neural measures, relevant for researchers using language models to study human cognition.
The study tackled the problem of whether larger Transformer-based language models predict human brain activity less effectively, finding that as model perplexity increases, their surprisal estimates fit fMRI data more poorly, with results generalizing across three language families and two datasets.
As Transformers become more widely incorporated into natural language processing tasks, there has been considerable interest in using surprisal from these models as predictors of human sentence processing difficulty. Recent work has observed a positive relationship between Transformer-based models' perplexity and the predictive power of their surprisal estimates on reading times, showing that language models with more parameters and trained on more data are less predictive of human reading times. However, these studies focus on predicting latency-based measures (i.e., self-paced reading times and eye-gaze durations) with surprisal estimates from Transformer-based language models. This trend has not been tested on brain imaging data. This study therefore evaluates the predictive power of surprisal estimates from 17 pre-trained Transformer-based models across three different language families on two functional magnetic resonance imaging datasets. Results show that the positive relationship between model perplexity and model fit still obtains, suggesting that this trend is not specific to latency-based measures and can be generalized to neural measures.