Sparse Autoencoders Can Interpret Randomly Initialized Transformers
This raises questions about the reliability of SAEs for mechanistic interpretability in AI, suggesting potential limitations in distinguishing meaningful patterns from random noise.
The paper investigated whether sparse autoencoders (SAEs) can interpret randomly initialized transformers, finding that random and trained transformers produce similarly interpretable SAE latents with comparable quality metrics across model sizes and layers.
Sparse autoencoders (SAEs) are an increasingly popular technique for interpreting the internal representations of transformers. In this paper, we apply SAEs to 'interpret' random transformers, i.e., transformers where the parameters are sampled IID from a Gaussian rather than trained on text data. We find that random and trained transformers produce similarly interpretable SAE latents, and we confirm this finding quantitatively using an open-source auto-interpretability pipeline. Further, we find that SAE quality metrics are broadly similar for random and trained transformers. We find that these results hold across model sizes and layers. We discuss a number of number interesting questions that this work raises for the use of SAEs and auto-interpretability in the context of mechanistic interpretability.