Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts
For LLM researchers and practitioners, this provides a mechanistic understanding of model internals and a parameter-efficient fine-tuning method.
The paper identifies a sparse subset of 'keystone neurons' in LLMs whose removal collapses model behavior, and shows that fine-tuning only these neurons achieves comparable or better task performance than full fine-tuning while preserving other capabilities.
Large language models (LLMs) display strong comprehensive abilities, yet the internal mechanisms that support these behaviors remain insufficiently understood. In this work, we show that across a wide range of open-weight Transformers, a subset of neurons remains consistently highly activated during inference across tasks of multiple capability dimensions. By probing along the cross-task activation strength, an extremely sparse subset is isolated, whose removal causes a collapse in model behavior, which we term keystone neurons. Our analysis reveals that keystone neurons are a stable and intrinsic neuron subset of the model that is largely established during pretraining. The parameters associated with these neurons are tightly calibrated during the training process, and their precise values are critical for the capabilities of the model. Building on these insights, we propose a supervised fine-tuning approach that updates only keystone neurons, achieving task gains comparable to or even better than full-parameter fine-tuning while better preserving performance in other capability dimensions, despite modifying a much smaller number of parameters.