MPViT: Multi-Path Vision Transformer for Dense Prediction
This work addresses the need for effective multi-scale backbones in computer vision, offering a versatile solution for various dense prediction tasks, though it is incremental as it builds on existing transformer architectures.
The paper tackles the problem of multi-scale feature representation for dense vision tasks like object detection and segmentation by introducing MPViT, a multi-path vision transformer that embeds patches of different scales simultaneously and aggregates features, achieving superior performance over state-of-the-art vision transformers on ImageNet classification, object detection, instance segmentation, and semantic segmentation across models from 5M to 73M parameters.
Dense computer vision tasks such as object detection and segmentation require effective multi-scale feature representation for detecting or classifying objects or regions with varying sizes. While Convolutional Neural Networks (CNNs) have been the dominant architectures for such tasks, recently introduced Vision Transformers (ViTs) aim to replace them as a backbone. Similar to CNNs, ViTs build a simple multi-stage structure (i.e., fine-to-coarse) for multi-scale representation with single-scale patches. In this work, with a different perspective from existing Transformers, we explore multi-scale patch embedding and multi-path structure, constructing the Multi-Path Vision Transformer (MPViT). MPViT embeds features of the same size~(i.e., sequence length) with patches of different scales simultaneously by using overlapping convolutional patch embedding. Tokens of different scales are then independently fed into the Transformer encoders via multiple paths and the resulting features are aggregated, enabling both fine and coarse feature representations at the same feature level. Thanks to the diverse, multi-scale feature representations, our MPViTs scaling from tiny~(5M) to base~(73M) consistently achieve superior performance over state-of-the-art Vision Transformers on ImageNet classification, object detection, instance segmentation, and semantic segmentation. These extensive results demonstrate that MPViT can serve as a versatile backbone network for various vision tasks. Code will be made publicly available at \url{https://git.io/MPViT}.