Singular Vectors of Attention Heads Align with Features
This work addresses a foundational gap in mechanistic interpretability for AI researchers, providing theoretical justification for an assumed method, though it is incremental in validating existing approaches.
The paper tackles the problem of justifying when singular vectors of attention matrices align with feature representations in language models, showing through experiments and theory that such alignment occurs under certain conditions and can be used for feature identification.
Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made an implicit assumption that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this assumption is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in a manner consistent with predictions in real models. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.