Interpretable Vision Transformers in Monocular Depth Estimation via SVDA
This addresses the problem of interpretability in dense prediction models for applications like robotics and autonomous driving, though it is incremental as it builds on existing DPT architectures.
The paper tackled the opacity of self-attention mechanisms in Transformers for monocular depth estimation by introducing SVD-Inspired Attention (SVDA), which preserved or slightly improved accuracy on KITTI and NYU-v2 datasets while enabling interpretable attention maps through six spectral indicators.
Monocular depth estimation is a central problem in computer vision with applications in robotics, AR, and autonomous driving, yet the self-attention mechanisms that drive modern Transformer architectures remain opaque. We introduce SVD-Inspired Attention (SVDA) into the Dense Prediction Transformer (DPT), providing the first spectrally structured formulation of attention for dense prediction tasks. SVDA decouples directional alignment from spectral modulation by embedding a learnable diagonal matrix into normalized query-key interactions, enabling attention maps that are intrinsically interpretable rather than post-hoc approximations. Experiments on KITTI and NYU-v2 show that SVDA preserves or slightly improves predictive accuracy while adding only minor computational overhead. More importantly, SVDA unlocks six spectral indicators that quantify entropy, rank, sparsity, alignment, selectivity, and robustness. These reveal consistent cross-dataset and depth-wise patterns in how attention organizes during training, insights that remain inaccessible in standard Transformers. By shifting the role of attention from opaque mechanism to quantifiable descriptor, SVDA redefines interpretability in monocular depth estimation and opens a principled avenue toward transparent dense prediction models.