Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
This work addresses the challenge of fine-grained mechanistic interpretability for researchers studying transformer models, though it is incremental in advancing existing methods.
The authors tackled the problem of understanding internal computations in transformer-based language models by decomposing attention heads and MLPs into orthogonal singular directions, revealing superposed and independent subfunctions within single components, as validated on tasks like IOI, GP, and GT where canonical heads like name movers encode multiple overlapping subfunctions aligned with distinct directions.
Transformer-based language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood. Existing mechanistic interpretability methods typically treat attention heads and multilayer perceptron layers (MLPs) (the building blocks of a transformer architecture) as indivisible units, overlooking possibilities of functional substructure learned within them. In this work, we introduce a more fine-grained perspective that decomposes these components into orthogonal singular directions, revealing superposed and independent computations within a single head or MLP. We validate our perspective on widely used standard tasks like Indirect Object Identification (IOI), Gender Pronoun (GP), and Greater Than (GT), showing that previously identified canonical functional heads, such as the name mover, encode multiple overlapping subfunctions aligned with distinct singular directions. Nodes in a computational graph, that are previously identified as circuit elements show strong activation along specific low-rank directions, suggesting that meaningful computations reside in compact subspaces. While some directions remain challenging to interpret fully, our results highlight that transformer computations are more distributed, structured, and compositional than previously assumed. This perspective opens new avenues for fine-grained mechanistic interpretability and a deeper understanding of model internals.