Weight-sparse transformers have interpretable circuits
This addresses the challenge of mechanistic interpretability in AI for researchers seeking human-understandable insights into model behavior, but it is incremental as it builds on existing sparse training and pruning techniques.
The paper tackles the problem of finding interpretable circuits in language models by training weight-sparse transformers, where most weights are zero, to isolate circuits for hand-crafted tasks, resulting in circuits with neurons and connections that correspond to natural concepts, though scaling beyond tens of millions of nonzero parameters remains challenging.
Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be zeros, so that each neuron only has a few connections. To recover fine-grained circuits underlying each of several hand-crafted tasks, we prune the models to isolate the part responsible for the task. These circuits often contain neurons and residual channels that correspond to natural concepts, with a small number of straightforwardly interpretable connections between them. We study how these models scale and find that making weights sparser trades off capability for interpretability, and scaling model size improves the capability-interpretability frontier. However, scaling sparse models beyond tens of millions of nonzero parameters while preserving interpretability remains a challenge. In addition to training weight-sparse models de novo, we show preliminary results suggesting our method can also be adapted to explain existing dense models. Our work produces circuits that achieve an unprecedented level of human understandability and validates them with considerable rigor.