Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning
This addresses the interpretability challenge for researchers and practitioners in AI, though it is incremental as it builds on existing circuit analysis methods.
The paper tackled the problem of understanding the internal reasoning mechanisms of transformer-based language models, which are opaque due to their complexity, by using circuit analysis and self-influence functions to map reasoning paths in GPT-2 on a prediction task, revealing a human-interpretable process.
Transformer-based language models have achieved significant success; however, their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. While previous studies have demonstrated that these models implicitly embed reasoning trees, humans typically employ various distinct logical reasoning mechanisms to complete the same task. It is still unclear which multi-step reasoning mechanisms are used by language models to solve such tasks. In this paper, we aim to address this question by investigating the mechanistic interpretability of language models, particularly in the context of multi-step reasoning tasks. Specifically, we employ circuit analysis and self-influence functions to evaluate the changing importance of each token throughout the reasoning process, allowing us to map the reasoning paths adopted by the model. We apply this methodology to the GPT-2 model on a prediction task (IOI) and demonstrate that the underlying circuits reveal a human-interpretable reasoning process used by the model.