A Primer on the Inner Workings of Transformer-based Language Models
This primer synthesizes existing research for researchers and practitioners in AI interpretability, but it is incremental as it primarily reviews and organizes known methods rather than introducing new findings.
The authors tackled the need for contextualizing insights from interpreting Transformer-based language models by providing a concise technical introduction to current techniques, focusing on generative decoder-only architectures, and concluded with a comprehensive overview of known internal mechanisms and connections across approaches.
The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive overview of the known internal mechanisms implemented by these models, uncovering connections across popular approaches and active research directions in this area.