When Language Models Lose Their Mind: The Consequences of Brain Misalignment
This addresses the problem of understanding the role of brain alignment in AI language models for researchers and developers, offering novel insights into neural representations and linguistic processing.
The study investigated the impact of brain alignment on large language models by creating brain-misaligned models that predict brain activity poorly while maintaining language modeling performance, and found that this misalignment substantially impairs performance on over 200 downstream linguistic tasks.
While brain-aligned large language models (LLMs) have garnered attention for their potential as cognitive models and for potential for enhanced safety and trustworthiness in AI, the role of this brain alignment for linguistic competence remains uncertain. In this work, we investigate the functional implications of brain alignment by introducing brain-misaligned models--LLMs intentionally trained to predict brain activity poorly while maintaining high language modeling performance. We evaluate these models on over 200 downstream tasks encompassing diverse linguistic domains, including semantics, syntax, discourse, reasoning, and morphology. By comparing brain-misaligned models with well-matched brain-aligned counterparts, we isolate the specific impact of brain alignment on language understanding. Our experiments reveal that brain misalignment substantially impairs downstream performance, highlighting the critical role of brain alignment in achieving robust linguistic competence. These findings underscore the importance of brain alignment in LLMs and offer novel insights into the relationship between neural representations and linguistic processing.