Neuron Empirical Gradient: Discovering and Quantifying Neurons Global Linear Controllability
This addresses the problem of limited progress in knowledge editing for researchers by providing a method to analyze neuron behavior at scale, though it is incremental as it builds on existing neuron intervention techniques.
The paper tackles the unclear broader role of neuron activations in pre-trained language models by uncovering a global linear relationship between activations and outputs, quantified as neuron empirical gradient (NEG), and shows it effectively represents model knowledge on benchmarks like MCEval8k.
While feed-forward neurons in pre-trained language models (PLMs) can encode knowledge, past research targeted a small subset of neurons that heavily influence outputs. This leaves the broader role of neuron activations unclear, limiting progress in areas like knowledge editing. We uncover a global linear relationship between neuron activations and outputs using neuron interventions on a knowledge probing dataset. The gradient of this linear relationship, which we call the neuron empirical gradient (NEG), captures how changes in activations affect predictions. To compute NEG efficiently, we propose NeurGrad, enabling large-scale analysis of neuron behavior in PLMs. We also show that NEG effectively captures language skills across diverse prompts through skill neuron probing. Experiments on MCEval8k, a multi-genre multiple-choice knowledge benchmark, support NEG's ability to represent model knowledge. Further analysis highlights the key properties of NEG-based skill representation: efficiency, robustness, flexibility, and interdependency. The code and data are released.