Local-to-Global Self-Attention in Vision Transformers
This work addresses efficiency and performance issues in vision transformers for computer vision tasks, representing an incremental advancement.
The paper tackles the lack of global feature reasoning in hierarchical vision transformers by proposing a multi-path structure that enables local-to-global reasoning at multiple granularities, achieving notable improvements in image classification and semantic segmentation with marginal computational overhead.
Transformers have demonstrated great potential in computer vision tasks. To avoid dense computations of self-attentions in high-resolution visual data, some recent Transformer models adopt a hierarchical design, where self-attentions are only computed within local windows. This design significantly improves the efficiency but lacks global feature reasoning in early stages. In this work, we design a multi-path structure of the Transformer, which enables local-to-global reasoning at multiple granularities in each stage. The proposed framework is computationally efficient and highly effective. With a marginal increasement in computational overhead, our model achieves notable improvements in both image classification and semantic segmentation. Code is available at https://github.com/ljpadam/LG-Transformer