DMFormer: Closing the Gap Between CNN and Vision Transformers
This work addresses efficiency and performance issues in computer vision models for researchers and practitioners, though it is incremental as it builds on existing transformer and CNN architectures.
The paper tackles the computational cost and performance gap between vision transformers and CNNs by proposing DMFormer, which replaces self-attention with a Dynamic Multi-level Attention mechanism, achieving state-of-the-art results on ImageNet-1K and ADE20K datasets.
Vision transformers have shown excellent performance in computer vision tasks. As the computation cost of their self-attention mechanism is expensive, recent works tried to replace the self-attention mechanism in vision transformers with convolutional operations, which is more efficient with built-in inductive bias. However, these efforts either ignore multi-level features or lack dynamic prosperity, leading to sub-optimal performance. In this paper, we propose a Dynamic Multi-level Attention mechanism (DMA), which captures different patterns of input images by multiple kernel sizes and enables input-adaptive weights with a gating mechanism. Based on DMA, we present an efficient backbone network named DMFormer. DMFormer adopts the overall architecture of vision transformers, while replacing the self-attention mechanism with our proposed DMA. Extensive experimental results on ImageNet-1K and ADE20K datasets demonstrated that DMFormer achieves state-of-the-art performance, which outperforms similar-sized vision transformers(ViTs) and convolutional neural networks (CNNs).