Decision-Aware Attention Propagation for Vision Transformer Explainability
For researchers and practitioners needing interpretable Vision Transformers, DAP improves explanation quality by combining attention propagation with decision-relevant priors, though it is an incremental improvement over existing methods.
Existing attention-based explanation methods for Vision Transformers lack class discriminability, while gradient-based methods do not exploit hierarchical attention propagation. The proposed Decision-Aware Attention Propagation (DAP) integrates gradient-based token importance into attention rollout, producing attribution maps that are more class-sensitive, compact, and faithful, consistently outperforming baselines across ViT variants.
Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet their prediction process remains difficult to interpret because information is propagated through complex interactions across layers and attention heads. Existing attention based explanation methods provide an intuitive way to trace information flow. However, they rely mainly on raw attention weights, which do not explicitly reflect the final decision and often lead to explanations with limited class discriminability. In contrast, gradient based localization methods are more effective at highlighting class specific evidence, but they do not fully exploit the hierarchical attention propagation mechanism of transformers. To address this limitation, we propose Decision-Aware Attention Propagation (DAP), an attribution method that injects decision-relevant priors into transformer attention propagation. By estimating token importance through gradient based localization and integrating it into layer wise attention rollout, the method captures both the structural flow of attention and the evidence most relevant to the final prediction. Consequently, DAP produces attribution maps that are more class sensitive, compact, and faithful than those generated by conventional attention based methods. Extensive experiments across Vision Transformer variants of different model scales show that DAP consistently outperforms existing baselines in both quantitative metrics and qualitative visualizations, indicating that decision aware propagation is an effective direction for improving ViT interpretability.