Less Memory, Faster Speed: Refining Self-Attention Module for Image Reconstruction
This addresses a bottleneck for researchers and practitioners in image processing by enabling more efficient application of self-attention to higher-resolution images, though it is incremental as it builds on existing SAGAN methods.
The paper tackled the high time and space complexity of self-attention modules in image reconstruction by refining the module to reduce complexity from O(n^2) to O(n) while maintaining equivalent performance, achieving comparable effectiveness with faster speed and less memory usage in experiments on benchmark datasets.
Self-attention (SA) mechanisms can capture effectively global dependencies in deep neural networks, and have been applied to natural language processing and image processing successfully. However, SA modules for image reconstruction have high time and space complexity, which restrict their applications to higher-resolution images. In this paper, we refine the SA module in self-attention generative adversarial networks (SAGAN) via adapting a non-local operation, revising the connectivity among the units in SA module and re-implementing its computational pattern, such that its time and space complexity is reduced from $\text{O}(n^2)$ to $\text{O}(n)$, but it is still equivalent to the original SA module. Further, we explore the principles behind the module and discover that our module is a special kind of channel attention mechanisms. Experimental results based on two benchmark datasets of image reconstruction, verify that under the same computational environment, two models can achieve comparable effectiveness for image reconstruction, but the proposed one runs faster and takes up less memory space.