Auxiliary Metrics Help Decoding Skill Neurons in the Wild
This work addresses interpretability for AI researchers, but it is incremental as it builds upon prior skill neuron identification methods.
The paper tackled the problem of interpreting internal mechanisms in large language models by introducing a method to isolate neurons encoding specific skills, demonstrating its ability to detect neurons that drive known skills and reveal shortcuts in arithmetic reasoning on BigBench.
Large language models (LLMs) exhibit remarkable capabilities across a wide range of tasks, yet their internal mechanisms remain largely opaque. In this paper, we introduce a simple, lightweight, and broadly applicable method with a focus on isolating neurons that encode specific skills. Building upon prior work that identified "skill neurons" via soft prompt training on classification tasks, our approach extends the analysis to complex scenarios involving multiple skills. We correlate neuron activations with auxiliary metrics -- such as external labels and the model's own confidence score -- thereby uncovering interpretable and task-specific behaviors without the need for manual token aggregation. We empirically validate our method on tasks spanning open-ended text generation and natural language inference, demonstrating its ability to detect neurons that not only drive known skills but also reveal previously unidentified shortcuts in arithmetic reasoning on BigBench.