$E(2)$-Equivariant Vision Transformer
This addresses a fundamental limitation in computer vision models for researchers and practitioners, though it appears incremental as it builds on prior equivariant ViT attempts.
The paper tackles the problem of positional encoding in Vision Transformers hindering equivariance learning, resulting in a Group Equivariant Vision Transformer (GE-ViT) that significantly outperforms non-equivariant self-attention networks on standard benchmarks.
Vision Transformer (ViT) has achieved remarkable performance in computer vision. However, positional encoding in ViT makes it substantially difficult to learn the intrinsic equivariance in data. Initial attempts have been made on designing equivariant ViT but are proved defective in some cases in this paper. To address this issue, we design a Group Equivariant Vision Transformer (GE-ViT) via a novel, effective positional encoding operator. We prove that GE-ViT meets all the theoretical requirements of an equivariant neural network. Comprehensive experiments are conducted on standard benchmark datasets, demonstrating that GE-ViT significantly outperforms non-equivariant self-attention networks. The code is available at https://github.com/ZJUCDSYangKaifan/GEVit.