Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints
This provides a foundational explanation for interpretability in AI, addressing a core theoretical gap for researchers and practitioners.
The paper tackles the problem of why linear interpretability methods succeed in deep nonlinear transformers by showing it is a consequence of architectural constraints, leading to a theorem that enables zero-shot identification of semantic structure without labeled data, validated empirically across tasks and models.
Linear probes and sparse autoencoders consistently recover meaningful structure from transformer representations -- yet why should such simple methods succeed in deep, nonlinear systems? We show this is not merely an empirical regularity but a consequence of architectural necessity: transformers communicate information through linear interfaces (attention OV circuits, unembedding matrices), and any semantic feature decoded through such an interface must occupy a context-invariant linear subspace. We formalize this as the \emph{Invariant Subspace Necessity} theorem and derive the \emph{Self-Reference Property}: tokens directly provide the geometric direction for their associated features, enabling zero-shot identification of semantic structure without labeled data or learned probes. Empirical validation in eight classification tasks and four model families confirms the alignment between class tokens and semantically related instances. Our framework provides \textbf{a principled architectural explanation} for why linear interpretability methods work, unifying linear probes and sparse autoencoders.