RegionViT: Regional-to-Local Attention for Vision Transformers
This work addresses the inefficiency of global self-attention in Vision Transformers for computer vision applications, offering an incremental improvement over existing ViT variants.
The paper tackles the suboptimal architecture of vanilla Vision Transformers for vision tasks by proposing RegionViT, which uses a pyramid structure and regional-to-local attention to combine global and local information. The model achieves state-of-the-art or competitive performance on four vision tasks, including image classification, object detection, semantic segmentation, and action recognition.
Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional self-attention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on four vision tasks, including image classification, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models are available at https://github.com/ibm/regionvit.