Sparse Reward Subsystem in Large Language Models
This work provides insights into the internal mechanisms of LLMs, potentially aiding in model interpretability and optimization for AI researchers.
The paper identifies a sparse reward subsystem in LLMs, analogous to the brain's reward system, containing value neurons that represent internal state value expectations and are crucial for reasoning, with experiments showing robustness across datasets, scales, and architectures, and transferability across models.
In this paper, we identify a sparse reward subsystem within the hidden states of Large Language Models (LLMs), drawing an analogy to the biological reward subsystem in the human brain. We demonstrate that this subsystem contains value neurons that represent the model's internal expectation of state value, and through intervention experiments, we establish the importance of these neurons for reasoning. Our experiments reveal that these value neurons are robust across diverse datasets, model scales, and architectures; furthermore, they exhibit significant transferability across different datasets and models fine-tuned from the same base model. By examining cases where value predictions and actual rewards diverge, we identify dopamine neurons within the reward subsystem which encode reward prediction errors (RPE). These neurons exhibit high activation when the reward is higher than expected and low activation when the reward is lower than expected.