Mapping Brains with Language Models: A Survey
This survey addresses the problem of synthesizing evidence for brain-language model mappings for researchers in neuroscience and AI, but it is incremental as it reviews existing studies without new findings.
The paper surveys over 30 studies to evaluate evidence for structural similarities between brain and language model activations, finding that accumulated evidence remains ambiguous but correlations with model size and quality offer cautious optimism.
Over the years, many researchers have seemingly made the same observation: Brain and language model activations exhibit some structural similarities, enabling linear partial mappings between features extracted from neural recordings and computational language models. In an attempt to evaluate how much evidence has been accumulated for this observation, we survey over 30 studies spanning 10 datasets and 8 metrics. How much evidence has been accumulated, and what, if anything, is missing before we can draw conclusions? Our analysis of the evaluation methods used in the literature reveals that some of the metrics are less conservative. We also find that the accumulated evidence, for now, remains ambiguous, but correlations with model size and quality provide grounds for cautious optimism.