What Helps -- and What Hurts: Bidirectional Explanations for Vision Transformers
This work addresses the interpretability challenge for ViTs, which is crucial for users in computer vision, but it is incremental as it builds on prior CAM-based approaches.
The paper tackles the problem of interpreting Vision Transformers (ViTs) by proposing BiCAM, a bidirectional class activation mapping method that captures both positive and negative contributions to predictions, improving localization and faithfulness across ImageNet, VOC, and COCO datasets.
Vision Transformers (ViTs) achieve strong performance in visual recognition, yet their decision-making remains difficult to interpret. We propose BiCAM, a bidirectional class activation mapping method that captures both supportive (positive) and suppressive (negative) contributions to model predictions. Unlike prior CAM-based approaches that discard negative signals, BiCAM preserves signed attributions to produce more complete and contrastive explanations. BiCAM further introduces a Positive-to-Negative Ratio (PNR) that summarizes attribution balance and enables lightweight detection of adversarial examples without retraining. Across ImageNet, VOC, and COCO, BiCAM improves localization and faithfulness while remaining computationally efficient. It generalizes to multiple ViT variants, including DeiT and Swin. These results suggest the importance of modeling both supportive and suppressive evidence for interpreting transformer-based vision models.