Finding Neurons in a Haystack: Case Studies with Sparse Probing
This work addresses the interpretability of LLMs for researchers and practitioners, providing insights into feature representation, but it is incremental as it builds on existing probing methods.
The authors tackled the problem of understanding how high-level human-interpretable features are represented in large language models (LLMs) by training sparse linear classifiers on internal neuron activations, finding that early layers use sparse neuron combinations, middle layers have dedicated neurons, and increasing model scale generally increases representational sparsity across over 100 features in models from 70 million to 6.9 billion parameters.
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.