Uncovering Intermediate Variables in Transformers using Circuit Probing
This provides a unified analysis tool for researchers to interpret complex models, though it is incremental in extending existing methods.
The authors tackled the problem of interpreting neural network algorithms by proposing circuit probing, a technique that automatically uncovers low-level circuits computing hypothesized intermediate variables, and demonstrated its effectiveness on arithmetic tasks and real-world language models like GPT2-Small and Medium for tasks such as subject-verb agreement.
Neural network models have achieved high performance on a wide variety of complex tasks, but the algorithms that they implement are notoriously difficult to interpret. It is often necessary to hypothesize intermediate variables involved in a network's computation in order to understand these algorithms. For example, does a language model depend on particular syntactic properties when generating a sentence? Yet, existing analysis tools make it difficult to test hypotheses of this type. We propose a new analysis technique - circuit probing - that automatically uncovers low-level circuits that compute hypothesized intermediate variables. This enables causal analysis through targeted ablation at the level of model parameters. We apply this method to models trained on simple arithmetic tasks, demonstrating its effectiveness at (1) deciphering the algorithms that models have learned, (2) revealing modular structure within a model, and (3) tracking the development of circuits over training. Across these three experiments we demonstrate that circuit probing combines and extends the capabilities of existing methods, providing one unified approach for a variety of analyses. Finally, we demonstrate circuit probing on a real-world use case: uncovering circuits that are responsible for subject-verb agreement and reflexive anaphora in GPT2-Small and Medium.