A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation
This work addresses the problem of simplifying ViT architectures for computer vision practitioners, but it is incremental as it builds on existing ViT designs without introducing a new paradigm.
The authors tackled the challenge of adapting Vision Transformers (ViTs) to object detection and instance segmentation by showing that a simple, single-scale ViT design can achieve strong performance without complex multistage architectures. Their proposed Universal Vision Transformer (UViT) achieved competitive results on COCO benchmarks, though no specific numbers were provided in the abstract.
This work presents a simple vision transformer design as a strong baseline for object localization and instance segmentation tasks. Transformers recently demonstrate competitive performance in image classification tasks. To adopt ViT to object detection and dense prediction tasks, many works inherit the multistage design from convolutional networks and highly customized ViT architectures. Behind this design, the goal is to pursue a better trade-off between computational cost and effective aggregation of multiscale global contexts. However, existing works adopt the multistage architectural design as a black-box solution without a clear understanding of its true benefits. In this paper, we comprehensively study three architecture design choices on ViT -- spatial reduction, doubled channels, and multiscale features -- and demonstrate that a vanilla ViT architecture can fulfill this goal without handcrafting multiscale features, maintaining the original ViT design philosophy. We further complete a scaling rule to optimize our model's trade-off on accuracy and computation cost / model size. By leveraging a constant feature resolution and hidden size throughout the encoder blocks, we propose a simple and compact ViT architecture called Universal Vision Transformer (UViT) that achieves strong performance on COCO object detection and instance segmentation tasks.