Sparse Autoencoders Trained on the Same Data Learn Different Features
This is an incremental finding that addresses interpretability issues for researchers using SAEs to analyze large language models.
The research tackled the problem of sparse autoencoders (SAEs) identifying inconsistent features when trained on the same model and data with different random seeds, finding that only 30% of features were shared across seeds in an example with 131K latents. The result suggests SAEs provide a pragmatic decomposition rather than a universal feature set.
Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true underlying features used by a model, our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features. For example, in an SAE with 131K latents trained on a feedforward network in Llama 3 8B, only 30% of the features were shared across different seeds. We observed this phenomenon across multiple layers of three different LLMs, two datasets, and several SAE architectures. While ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity. Our results suggest that the set of features uncovered by an SAE should be viewed as a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features "truly used" by the model.