Building Better Activation Oracles
For researchers in mechanistic interpretability, this work provides incremental improvements to AOs and a new evaluation benchmark, but the gains are marginal.
This paper improves Activation Oracles (AOs) for interpreting residual stream activations by addressing issues like hallucinations and vagueness through four training improvements, and introduces AObench, the first comprehensive evaluation suite for AO quality. The capability improvements are marginal, but quality-of-life improvements are substantial.
Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Additionally, text-inversion confounds make them hard to evaluate. To this end, we improve the Activation Oracle (AO) training regime in four ways: training on on-policy rollouts, improving the conversational dataset, feeding more layers and an improvement to the injection formula. The capability improvements are marginal, but quality of life improvements are quite substantial. In addition, we open source the first comprehensive evaluation suite for AO quality, which we call AObench. Overall, we hope that our work sets a foundation that helps improve AOs and other models in the paradigm of scalable, end-to-end interpretability.