Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution
This work addresses the scalability bottleneck of SAE-based interpretability for long-context language models by introducing a turn-level aggregation method.
The authors propose turn-averaged SAEs that compress per-token activations into fixed-length turn-level representations, enabling interpretability at long context lengths. They show these features describe turn-level characteristics more completely than per-token features and simplify downstream tasks like attribution graphs.
Sparse autoencoders (SAEs) have become a useful tool for extracting interpretable features in language models. However, standard SAE architectures operate on individual token activations, meaning that the number of active features scales linearly with context length, and studying long model transcripts becomes difficult. We introduce turn-averaged SAEs, which represent a single Human or Assistant turn with a fixed number of features by learning to reconstruct the average model activation across the turn. We find that turn-averaged features describe a single turn's high-level characteristics more completely than per-token features when judged by an LLM. We also demonstrate that turn-averaged SAEs greatly simplify common downstream uses of SAEs like attribution graphs. Broadly, turn-averaged SAEs make interpretability techniques practical at long context lengths.