Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer
This work addresses the need for more faithful explanations of Vision Transformers in computer vision, offering an incremental improvement over prior methods that only considered attention weights.
The paper tackles the problem of inaccurate post-hoc explanations for Vision Transformers by proposing TokenTM, a method that incorporates token transformation effects alongside attention weights, achieving superior performance in segmentation and perturbation tests compared to existing methods.
While Transformers have rapidly gained popularity in various computer vision applications, post-hoc explanations of their internal mechanisms remain largely unexplored. Vision Transformers extract visual information by representing image regions as transformed tokens and integrating them via attention weights. However, existing post-hoc explanation methods merely consider these attention weights, neglecting crucial information from the transformed tokens, which fails to accurately illustrate the rationales behind the models' predictions. To incorporate the influence of token transformation into interpretation, we propose TokenTM, a novel post-hoc explanation method that utilizes our introduced measurement of token transformation effects. Specifically, we quantify token transformation effects by measuring changes in token lengths and correlations in their directions pre- and post-transformation. Moreover, we develop initialization and aggregation rules to integrate both attention weights and token transformation effects across all layers, capturing holistic token contributions throughout the model. Experimental results on segmentation and perturbation tests demonstrate the superiority of our proposed TokenTM compared to state-of-the-art Vision Transformer explanation methods.