A Generic Shared Attention Mechanism for Various Backbone Neural Networks
This work addresses inefficiencies in self-attention mechanisms for deep learning practitioners, offering a parameter-efficient solution that enhances performance in multiple domains, though it is incremental as it builds on existing self-attention paradigms.
The paper tackled the problem of suboptimal performance and high parameter consumption in self-attention mechanisms by proposing Dense-and-Implicit Attention (DIA), which shares attention modules across layers and uses an LSTM for calibration, achieving consistent improvements across various backbone networks and tasks.
The self-attention mechanism has emerged as a critical component for improving the performance of various backbone neural networks. However, current mainstream approaches individually incorporate newly designed self-attention modules (SAMs) into each layer of the network for granted without fully exploiting their parameters' potential. This leads to suboptimal performance and increased parameter consumption as the network depth increases. To improve this paradigm, in this paper, we first present a counterintuitive but inherent phenomenon: SAMs tend to produce strongly correlated attention maps across different layers, with an average Pearson correlation coefficient of up to 0.85. Inspired by this inherent observation, we propose Dense-and-Implicit Attention (DIA), which directly shares SAMs across layers and employs a long short-term memory module to calibrate and bridge the highly correlated attention maps of different layers, thus improving the parameter utilization efficiency of SAMs. This design of DIA is also consistent with the neural network's dynamical system perspective. Through extensive experiments, we demonstrate that our simple yet effective DIA can consistently enhance various network backbones, including ResNet, Transformer, and UNet, across tasks such as image classification, object detection, and image generation using diffusion models.