Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers
This work challenges the necessity of architectural positional injection in Vision Transformers, offering a training-time alternative that achieves superior performance across multiple vision tasks.
The paper proposes Active Spatial Guidance (Guidance), a training-only objective that eliminates injected positional mechanisms in Vision Transformers by applying an auxiliary 2D coordinate-regression loss to final-layer patch tokens. Guidance consistently outperforms strong injected baselines on ImageNet-100 classification, ADE20K segmentation, and Hypersim depth estimation, and improves robustness under resolution transfer.
Vision Transformers (ViTs) commonly rely on injected positional mechanisms to address self-attention's permutation invariance. Motivated by the spatial regularities of natural images, we ask whether spatial organization can be induced from data rather than explicitly injected. Under controlled, matched from-scratch training, we propose Active Spatial Guidance (Guidance), a training-only objective that disables positional injection and applies an auxiliary 2D coordinate-regression loss to the final-layer patch tokens. The guidance head is used only during training and removed for inference; the deployed model consists of a positional-injection-free ViT encoder and the task-specific prediction module. Using DINOv3 ViT backbones, Guidance consistently improves performance on ImageNet-100 classification, ADE20K semantic segmentation, and Hypersim monocular depth estimation, outperforming strong injected baselines such as learned absolute positional embeddings and rotary positional embeddings under identical training protocols. On ImageNet-100, broader comparisons against representative injected positional designs further support Guidance's effectiveness. Guidance also improves robustness under resolution transfer, and multi-resolution training further strengthens accuracy across input sizes. Overall, our results suggest that spatial inductive bias in ViTs need not be architecturally injected, but can be shaped through training-time supervision. The code used for training and evaluation is publicly available in https://github.com/cloudlc/asg.