O-ViT: Orthogonal Vision Transformer
This addresses a specific problem in vision transformers for image recognition, offering an incremental improvement.
The paper tackles the scale ambiguity issue in Vision Transformers (ViT) caused by scaled dot-product self-attention, proposing O-ViT to optimize ViT from a geometric perspective by limiting parameters to an orthogonal manifold, which boosts performance by up to 3.6% on image recognition tasks.
Inspired by the tremendous success of the self-attention mechanism in natural language processing, the Vision Transformer (ViT) creatively applies it to image patch sequences and achieves incredible performance. However, the scaled dot-product self-attention of ViT brings about scale ambiguity to the structure of the original feature space. To address this problem, we propose a novel method named Orthogonal Vision Transformer (O-ViT), to optimize ViT from the geometric perspective. O-ViT limits parameters of self-attention blocks to be on the norm-keeping orthogonal manifold, which can keep the geometry of the feature space. Moreover, O-ViT achieves both orthogonal constraints and cheap optimization overhead by adopting a surjective mapping between the orthogonal group and its Lie algebra.We have conducted comparative experiments on image recognition tasks to demonstrate O-ViT's validity and experiments show that O-ViT can boost the performance of ViT by up to 3.6%.