ECViT: Efficient Convolutional Vision Transformer with Local-Attention and Multi-scale Stages
This work addresses efficiency challenges in computer vision for applications prioritizing performance without high resource use, representing an incremental improvement over existing hybrid models.
The paper tackles the high computational costs and data requirements of Vision Transformers by proposing ECViT, a hybrid CNN-Transformer architecture that achieves state-of-the-art performance on image classification tasks with low computational and storage demands.
Vision Transformers (ViTs) have revolutionized computer vision by leveraging self-attention to model long-range dependencies. However, ViTs face challenges such as high computational costs due to the quadratic scaling of self-attention and the requirement of a large amount of training data. To address these limitations, we propose the Efficient Convolutional Vision Transformer (ECViT), a hybrid architecture that effectively combines the strengths of CNNs and Transformers. ECViT introduces inductive biases such as locality and translation invariance, inherent to Convolutional Neural Networks (CNNs) into the Transformer framework by extracting patches from low-level features and enhancing the encoder with convolutional operations. Additionally, it incorporates local-attention and a pyramid structure to enable efficient multi-scale feature extraction and representation. Experimental results demonstrate that ECViT achieves an optimal balance between performance and efficiency, outperforming state-of-the-art models on various image classification tasks while maintaining low computational and storage requirements. ECViT offers an ideal solution for applications that prioritize high efficiency without compromising performance.