A Lightweight Convolution and Vision Transformer integrated model with Multi-scale Self-attention Mechanism
This provides a lightweight solution for fundamental vision tasks, addressing efficiency and performance trade-offs for real-world applications, but it is incremental as it builds on existing ViT and convolution methods.
The paper tackles the problem of Vision Transformers being computationally heavy and weak in local feature modeling by proposing SAEViT, an efficient model integrating sparse attention and convolution blocks, achieving Top-1 accuracies of 76.3% and 79.6% on ImageNet-1K with only 0.8 and 1.3 GFLOPs.
Vision Transformer (ViT) has prevailed in computer vision tasks due to its strong long-range dependency modelling ability. \textcolor{blue}{However, its large model size and weak local feature modeling ability hinder its application in real scenarios. To balance computation efficiency and performance in downstream vision tasks, we propose an efficient ViT model with sparse attention (dubbed SAEViT) and convolution blocks. Specifically, a Sparsely Aggregated Attention (SAA) module has been proposed to perform adaptive sparse sampling and recover the feature map via deconvolution operation,} which significantly reduces the computational complexity of attention operations. In addition, a Channel-Interactive Feed-Forward Network (CIFFN) layer is developed to enhance inter-channel information exchange through feature decomposition and redistribution, which mitigates the redundancy in traditional feed-forward networks (FFN). Finally, a hierarchical pyramid structure with embedded depth-wise separable convolutional blocks (DWSConv) is devised to further strengthen convolutional features. Extensive experiments on mainstream datasets show that SAEViT achieves Top-1 accuracies of 76.3\% and 79.6\% on the ImageNet-1K classification task with only 0.8 GFLOPs and 1.3 GFLOPs, respectively, demonstrating a lightweight solution for fundamental vision tasks.