MIB: A Mechanistic Interpretability Benchmark
This provides a standardized benchmark for researchers in mechanistic interpretability to compare methods, though it is incremental as it builds on existing evaluation needs without introducing new paradigms.
The authors tackled the lack of evaluation standards for mechanistic interpretability methods by proposing MIB, a benchmark with two tracks across tasks and models, finding that attribution and mask optimization methods excel in circuit localization, while supervised DAS outperforms in causal variable localization.
How can we know whether new mechanistic interpretability methods achieve real improvements? In pursuit of lasting evaluation standards, we propose MIB, a Mechanistic Interpretability Benchmark, with two tracks spanning four tasks and five models. MIB favors methods that precisely and concisely recover relevant causal pathways or causal variables in neural language models. The circuit localization track compares methods that locate the model components - and connections between them - most important for performing a task (e.g., attribution patching or information flow routes). The causal variable localization track compares methods that featurize a hidden vector, e.g., sparse autoencoders (SAEs) or distributed alignment search (DAS), and align those features to a task-relevant causal variable. Using MIB, we find that attribution and mask optimization methods perform best on circuit localization. For causal variable localization, we find that the supervised DAS method performs best, while SAE features are not better than neurons, i.e., non-featurized hidden vectors. These findings illustrate that MIB enables meaningful comparisons, and increases our confidence that there has been real progress in the field.