Switchable Self-attention Module
This work addresses inefficiencies in attention mechanism design for vision recognition, offering a more automated approach, though it is incremental in nature.
The authors tackled the problem of manually selecting and applying attention operators across different network layers by proposing a self-attention module (SEM) that automatically selects and integrates operators based on input information, achieving improved performance on benchmark datasets.
Attention mechanism has gained great success in vision recognition. Many works are devoted to improving the effectiveness of attention mechanism, which finely design the structure of the attention operator. These works need lots of experiments to pick out the optimal settings when scenarios change, which consumes a lot of time and computational resources. In addition, a neural network often contains many network layers, and most studies often use the same attention module to enhance different network layers, which hinders the further improvement of the performance of the self-attention mechanism. To address the above problems, we propose a self-attention module SEM. Based on the input information of the attention module and alternative attention operators, SEM can automatically decide to select and integrate attention operators to compute attention maps. The effectiveness of SEM is demonstrated by extensive experiments on widely used benchmark datasets and popular self-attention networks.