Linear Context Transform Block
This work addresses a specific bottleneck in attention-based models for computer vision tasks like image classification and object detection, offering an incremental improvement over existing methods.
The paper tackles the problem of understanding and improving the Squeeze-and-Excitation (SE) block's channel attention mechanism by proposing a Linear Context Transform (LCT) block, which groups channels and normalizes context features to reduce interference, resulting in performance gains such as 1.5-1.7% AP improvements on COCO object detection.
Squeeze-and-Excitation (SE) block presents a channel attention mechanism for modeling global context via explicitly capturing dependencies across channels. However, we are still far from understanding how the SE block works. In this work, we first revisit the SE block, and then present a detailed empirical study of the relationship between global context and attention distribution, based on which we propose a simple yet effective module, called Linear Context Transform (LCT) block. We divide all channels into different groups and normalize the globally aggregated context features within each channel group, reducing the disturbance from irrelevant channels. Through linear transform of the normalized context features, we model global context for each channel independently. The LCT block is extremely lightweight and easy to be plugged into different backbone models while with negligible parameters and computational burden increase. Extensive experiments show that the LCT block outperforms the SE block in image classification task on the ImageNet and object detection/segmentation on the COCO dataset with different backbone models. Moreover, LCT yields consistent performance gains over existing state-of-the-art detection architectures, e.g., 1.5$\sim$1.7% AP$^{bbox}$ and 1.0$\sim$1.2% AP$^{mask}$ improvements on the COCO benchmark, irrespective of different baseline models of varied capacities. We hope our simple yet effective approach will shed some light on future research of attention-based models.