Transformer Interpretability Beyond Attention Visualization
This work provides an improved interpretability method for researchers and practitioners working with Transformer networks, which are widely used in text and vision tasks, by offering a more accurate way to understand model decisions.
This paper addresses the problem of interpreting Transformer networks by proposing a novel method to compute and propagate relevancy scores through the layers, including attention layers and skip connections. The method, based on Deep Taylor Decomposition, is shown to maintain total relevancy across layers and demonstrates a clear advantage over existing explainability methods on visual Transformer networks and text classification.
Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.