Convolutional Xformers for Vision
This work addresses the problem of deploying efficient vision models for researchers and practitioners with constrained computational resources, though it appears incremental as it builds on existing linear attention and convolutional methods.
The authors tackled the limited practical use of vision transformers due to high computational costs and data requirements by proposing Convolutional X-formers for Vision (CXV), a linear attention-convolution hybrid architecture that reduces GPU usage and eliminates the need for class tokens and positional embeddings, resulting in outperforming other models in image classification with limited data and GPU resources.
Vision transformers (ViTs) have found only limited practical use in processing images, in spite of their state-of-the-art accuracy on certain benchmarks. The reason for their limited use include their need for larger training datasets and more computational resources compared to convolutional neural networks (CNNs), owing to the quadratic complexity of their self-attention mechanism. We propose a linear attention-convolution hybrid architecture -- Convolutional X-formers for Vision (CXV) -- to overcome these limitations. We replace the quadratic attention with linear attention mechanisms, such as Performer, Nyströmformer, and Linear Transformer, to reduce its GPU usage. Inductive prior for image data is provided by convolutional sub-layers, thereby eliminating the need for class token and positional embeddings used by the ViTs. We also propose a new training method where we use two different optimizers during different phases of training and show that it improves the top-1 image classification accuracy across different architectures. CXV outperforms other architectures, token mixers (e.g. ConvMixer, FNet and MLP Mixer), transformer models (e.g. ViT, CCT, CvT and hybrid Xformers), and ResNets for image classification in scenarios with limited data and GPU resources (cores, RAM, power).