Neuron Populations Exhibit Divergent Selectivity with Scale
For researchers studying neural network interpretability and scaling laws, this work reveals systematic changes in neuron-level structure with model scale, extending scaling laws beyond macroscopic observables.
The paper discovers that Rosetta Neurons (neurons with similar activation patterns across independently trained models) follow a sublinear power law in model size, growing in absolute number but shrinking as a fraction of total neurons, and become more selective and monosemantic with scale, a phenomenon termed the Neuron Polarization Effect.
We investigate whether neuron populations within neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss. To probe this question, we study Rosetta Neurons, a previously characterized class of neurons whose activation patterns are similar across independently trained models (Dravid et al., 2023). In separate analyses of language models up to 30B parameters and vision models up to 5B parameters, we observe that the population of Rosetta Neurons follows a sublinear power law in model size, growing in absolute number but occupying a shrinking fraction of the total neuron count. We further observe a Neuron Polarization Effect: Rosetta Neurons become more selective and increasingly monosemantic with scale, separating from a growing non-Rosetta population that remains less selective. An analytical model balancing feature utility against limited neuron capacity explains the sublinear power-law scaling and this polarization effect. Finally, we find that Rosetta Neurons become more domain-specialized with scale and illustrate their selectivity through a targeted data-filtering case study for continued pretraining. Our results point to a scaling law for interpretable, shared neuron-level structure, linking model size to systematic changes in neuron universality, selectivity, and specialization.