Adaptive Attention Span in Computer Vision
This work addresses efficiency and performance in vision tasks for researchers and practitioners, though it is incremental as it builds on existing local self-attention methods.
The paper tackles the problem of improving local self-attention kernels in computer vision by proposing a novel method to learn their size adaptively, resulting in performance gains compared to fixed-size attention and convolution kernels, with models showing fewer parameters and FLOPS.
Recent developments in Transformers for language modeling have opened new areas of research in computer vision. Results from late 2019 showed vast performance increases in both object detection and recognition when convolutions are replaced by local self-attention kernels. Models using local self-attention kernels were also shown to have less parameters and FLOPS compared to equivalent architectures that only use convolutions. In this work we propose a novel method for learning the local self-attention kernel size. We then compare its performance to fixed-size local attention and convolution kernels. The code for all our experiments and models is available at https://github.com/JoeRoussy/adaptive-attention-in-cv