Attention Flows for General Transformers
This work addresses the interpretability of Transformer models for researchers and practitioners, offering a tool to analyze token influence in NLP and reasoning tasks, but it is incremental as it builds on existing attention and Shapley value concepts.
The paper tackles the problem of quantifying how much each input token influences a Transformer model's prediction by formalizing a method to construct a flow network from attention values and extending it to general architectures, including auto-regressive decoders, and shows that running a maxflow algorithm yields Shapley values to compute token influence, with a library provided for computation and visualization.
In this paper, we study the computation of how much an input token in a Transformer model influences its prediction. We formalize a method to construct a flow network out of the attention values of encoder-only Transformer models and extend it to general Transformer architectures including an auto-regressive decoder. We show that running a maxflow algorithm on the flow network construction yields Shapley values, which determine the impact of a player in cooperative game theory. By interpreting the input tokens in the flow network as players, we can compute their influence on the total attention flow leading to the decoder's decision. Additionally, we provide a library that computes and visualizes the attention flow of arbitrary Transformer models. We show the usefulness of our implementation on various models trained on natural language processing and reasoning tasks.