Neuron-Level Knowledge Attribution in Large Language Models
This work addresses the challenge of neuron-level knowledge attribution for researchers aiming to understand model mechanisms, though it is incremental in improving existing static methods.
The paper tackles the problem of identifying important neurons for predictions in large language models, proposing a static method that outperforms seven others across three metrics and also identifies 'query neurons' that activate 'value neurons'.
Identifying important neurons for final predictions is essential for understanding the mechanisms of large language models. Due to computational constraints, current attribution techniques struggle to operate at neuron level. In this paper, we propose a static method for pinpointing significant neurons. Compared to seven other methods, our approach demonstrates superior performance across three metrics. Additionally, since most static methods typically only identify "value neurons" directly contributing to the final prediction, we propose a method for identifying "query neurons" which activate these "value neurons". Finally, we apply our methods to analyze six types of knowledge across both attention and feed-forward network (FFN) layers. Our method and analysis are helpful for understanding the mechanisms of knowledge storage and set the stage for future research in knowledge editing. The code is available on https://github.com/zepingyu0512/neuron-attribution.