Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability
This work addresses the problem of understanding how interpretable circuits in large language models generalize, which is important for researchers in mechanistic interpretability, though it is incremental as it builds on existing studies of a specific circuit.
The paper investigates the generalization of the indirect object identification circuit in GPT-2 small across different prompt formats, finding that it reuses all components and mechanisms with only minor additions, even in cases where the original algorithm should fail, as explained by a mechanism termed S2 Hacking.
Mechanistic interpretability aims to understand the inner workings of large neural networks by identifying circuits, or minimal subgraphs within the model that implement algorithms responsible for performing specific tasks. These circuits are typically discovered and analyzed using a narrowly defined prompt format. However, given the abilities of large language models (LLMs) to generalize across various prompt formats for the same task, it remains unclear how well these circuits generalize. For instance, it is unclear whether the models generalization results from reusing the same circuit components, the components behaving differently, or the use of entirely different components. In this paper, we investigate the generality of the indirect object identification (IOI) circuit in GPT-2 small, which is well-studied and believed to implement a simple, interpretable algorithm. We evaluate its performance on prompt variants that challenge the assumptions of this algorithm. Our findings reveal that the circuit generalizes surprisingly well, reusing all of its components and mechanisms while only adding additional input edges. Notably, the circuit generalizes even to prompt variants where the original algorithm should fail; we discover a mechanism that explains this which we term S2 Hacking. Our findings indicate that circuits within LLMs may be more flexible and general than previously recognized, underscoring the importance of studying circuit generalization to better understand the broader capabilities of these models.