GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
This addresses the issue of false sequential proximity in vision tasks for researchers and practitioners, though it is incremental as it builds on existing positional embedding methods.
The paper tackles the problem of standard Vision Transformers disrupting spatial topology by flattening 2D images into 1D sequences, and introduces Geometric Positional Embedding (GeoPE) to restore the 2D spatial manifold, resulting in consistent performance improvements in image classification, object detection, and 3D semantic segmentation.
Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.