Positional Label for Self-Supervised Vision Transformer
This work addresses the challenge of enhancing ViT performance for computer vision tasks, offering an incremental improvement through a plug-in self-supervised method.
The paper tackles the problem of improving vision transformers (ViTs) by proposing a self-supervised task where ViTs learn to recognize positional labels of image patches, resulting in performance gains such as 1.20% and 0.74% top-1 accuracy improvements on ImageNet for ViT-B and Swin-B, respectively.
Positional encoding is important for vision transformer (ViT) to capture the spatial structure of the input image. General effectiveness has been proven in ViT. In our work we propose to train ViT to recognize the positional label of patches of the input image, this apparently simple task actually yields a meaningful self-supervisory task. Based on previous work on ViT positional encoding, we propose two positional labels dedicated to 2D images including absolute position and relative position. Our positional labels can be easily plugged into various current ViT variants. It can work in two ways: (a) As an auxiliary training target for vanilla ViT (e.g., ViT-B and Swin-B) for better performance. (b) Combine the self-supervised ViT (e.g., MAE) to provide a more powerful self-supervised signal for semantic feature learning. Experiments demonstrate that with the proposed self-supervised methods, ViT-B and Swin-B gain improvements of 1.20% (top-1 Acc) and 0.74% (top-1 Acc) on ImageNet, respectively, and 6.15% and 1.14% improvement on Mini-ImageNet.