Language models align with brain regions that represent concepts across modalities
This research addresses the challenge of disentangling linguistic from conceptual representations in cognitive science and neuroscience, with implications for understanding how LMs might encode cross-modal meaning, though it is incremental in building on existing fMRI datasets and neural metrics.
The study investigated whether language models (LMs) align with brain regions that represent concepts consistently across different input modalities, such as sentences, word clouds, and images, using fMRI data. The results showed that both language-only and language-vision models better predicted brain signals in areas with higher meaning consistency, even when those areas were not strongly sensitive to language processing.
Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM--brain alignment and two neural metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) a novel measure of meaning consistency across input modalities, which quantifies how consistently a brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the signal better in more meaning-consistent areas of the brain, even when these areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning.