Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
This work addresses a key bottleneck in vision transformers for researchers and practitioners in computer vision, though it is incremental as it builds on existing window-based methods.
The authors tackled the problem of limited cross-window connections in Window-based Transformers for vision tasks by proposing Shuffle Transformer, which uses spatial shuffle and depth-wise convolution to enhance connections, achieving excellent performance on image classification, object detection, and semantic segmentation.
Very recently, Window-based Transformers, which computed self-attention within non-overlapping local windows, demonstrated promising results on image classification, semantic segmentation, and object detection. However, less study has been devoted to the cross-window connection which is the key element to improve the representation ability. In this work, we revisit the spatial shuffle as an efficient way to build connections among windows. As a result, we propose a new vision transformer, named Shuffle Transformer, which is highly efficient and easy to implement by modifying two lines of code. Furthermore, the depth-wise convolution is introduced to complement the spatial shuffle for enhancing neighbor-window connections. The proposed architectures achieve excellent performance on a wide range of visual tasks including image-level classification, object detection, and semantic segmentation. Code will be released for reproduction.