NAM: Normalization-based Attention Module
This work addresses model compression for deep learning practitioners by introducing an incremental improvement to attention modules for computational efficiency.
The paper tackles the problem of recognizing less salient features in attention mechanisms for model compression by proposing a normalization-based attention module (NAM) that suppresses less salient weights with a sparsity penalty, resulting in higher accuracy compared to three other attention mechanisms on Resnet and Mobilenet.
Recognizing less salient features is the key for model compression. However, it has not been investigated in the revolutionary attention mechanisms. In this work, we propose a novel normalization-based attention module (NAM), which suppresses less salient weights. It applies a weight sparsity penalty to the attention modules, thus, making them more computational efficient while retaining similar performance. A comparison with three other attention mechanisms on both Resnet and Mobilenet indicates that our method results in higher accuracy. Code for this paper can be publicly accessed at https://github.com/Christian-lyc/NAM.