Who Reasons in the Large Language Models?
This work addresses the problem of understanding and improving LLM interpretability for researchers and developers, offering insights that could lead to more efficient training strategies, though it is incremental in nature.
The paper investigates whether reasoning abilities in large language models (LLMs) are localized to specific modules, hypothesizing that the output projection module (oproj) in the Transformer's multi-head self-attention mechanism is primarily responsible for reasoning, while other modules handle fluent dialogue, as supported by diagnostic tools called Stethoscope for Networks (SfN).
Despite the impressive performance of large language models (LLMs), the process of endowing them with new capabilities--such as mathematical reasoning--remains largely empirical and opaque. A critical open question is whether reasoning abilities stem from the entire model, specific modules, or are merely artifacts of overfitting. In this work, we hypothesize that the reasoning capabilities in well-trained LLMs are primarily attributed to the output projection module (oproj) in the Transformer's multi-head self-attention (MHSA) mechanism. To support this hypothesis, we introduce Stethoscope for Networks (SfN), a suite of diagnostic tools designed to probe and analyze the internal behaviors of LLMs. Using SfN, we provide both circumstantial and empirical evidence suggesting that oproj plays a central role in enabling reasoning, whereas other modules contribute more to fluent dialogue. These findings offer a new perspective on LLM interpretability and open avenues for more targeted training strategies, potentially enabling more efficient and specialized LLMs.