Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning
This work clarifies the causal role of PEs in ViTs for computer vision, addressing a fundamental issue in model interpretability and performance.
The paper investigates how positional embeddings (PEs) in vision transformers (ViTs) act as geometric priors affecting spatial reasoning, showing through experiments on 14 models that PEs influence multi-view geometric consistency and spatial structure.
This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes