Are formal and functional linguistic mechanisms dissociated in language models?
This work addresses the problem of understanding and improving LLM capabilities for researchers and developers, but it is incremental as it builds on prior neuroscience-inspired localization ideas without achieving a unified formal network.
The paper investigated whether large language models (LLMs) exhibit distinct neural mechanisms for formal linguistic tasks (e.g., grammar) versus functional linguistic tasks (e.g., reasoning), by analyzing computational circuits across 5 models and 10 tasks. It found little overlap between formal and functional task circuits, and also little overlap among formal tasks themselves, though some separation in cross-task faithfulness suggests potential shared mechanisms.
Although large language models (LLMs) are increasingly capable, these capabilities are unevenly distributed: they excel at formal linguistic tasks, such as producing fluent, grammatical text, but struggle more with functional linguistic tasks like reasoning and consistent fact retrieval. Inspired by neuroscience, recent work suggests that to succeed on both formal and functional linguistic tasks, LLMs should use different mechanisms for each; such localization could either be built-in or emerge spontaneously through training. In this paper, we ask: do current models, with fast-improving functional linguistic abilities, exhibit distinct localization of formal and functional linguistic mechanisms? We answer this by finding and comparing the "circuits", or minimal computational subgraphs, responsible for various formal and functional tasks. Comparing 5 LLMs across 10 distinct tasks, we find that while there is indeed little overlap between circuits for formal and functional tasks, there is also little overlap between formal linguistic tasks, as exists in the human brain. Thus, a single formal linguistic network, unified and distinct from functional task circuits, remains elusive. However, in terms of cross-task faithfulness - the ability of one circuit to solve another's task - we observe a separation between formal and functional mechanisms, suggesting that shared mechanisms between formal tasks may exist.