Stand-Alone Self-Attention in Vision Models
This work addresses the need for more efficient and effective vision models for practitioners, though it is incremental as it builds on existing self-attention and ResNet architectures.
The paper tackled the problem of whether self-attention can replace convolutions as a stand-alone layer in vision models, and found that a pure self-attention model outperforms convolutional baselines on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters, and matches mAP on COCO object detection with 39% fewer FLOPS and 34% fewer parameters.
Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependencies. These efforts focus on augmenting convolutional models with content-based interactions, such as self-attention and non-local means, to achieve gains on a number of vision tasks. The natural question that arises is whether attention can be a stand-alone primitive for vision models instead of serving as just an augmentation on top of convolutions. In developing and testing a pure self-attention vision model, we verify that self-attention can indeed be an effective stand-alone layer. A simple procedure of replacing all instances of spatial convolutions with a form of self-attention applied to ResNet model produces a fully self-attentional model that outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters. On COCO object detection, a pure self-attention model matches the mAP of a baseline RetinaNet while having 39% fewer FLOPS and 34% fewer parameters. Detailed ablation studies demonstrate that self-attention is especially impactful when used in later layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox.