FasterViT: Fast Vision Transformers with Hierarchical Attention
This work addresses the problem of slow image processing in computer vision applications by providing a faster and more efficient transformer model, though it is incremental as it builds on existing CNN and ViT architectures.
The authors tackled the computational inefficiency of global self-attention in vision transformers by introducing FasterViT, a hybrid CNN-ViT model with Hierarchical Attention that reduces quadratic complexity, achieving state-of-the-art Pareto-front in accuracy and image throughput across classification, detection, and segmentation tasks.
We design a new family of hybrid CNN-ViT neural networks, named FasterViT, with a focus on high image throughput for computer vision (CV) applications. FasterViT combines the benefits of fast local representation learning in CNNs and global modeling properties in ViT. Our newly introduced Hierarchical Attention (HAT) approach decomposes global self-attention with quadratic complexity into a multi-level attention with reduced computational costs. We benefit from efficient window-based self-attention. Each window has access to dedicated carrier tokens that participate in local and global representation learning. At a high level, global self-attentions enable the efficient cross-window communication at lower costs. FasterViT achieves a SOTA Pareto-front in terms of accuracy and image throughput. We have extensively validated its effectiveness on various CV tasks including classification, object detection and segmentation. We also show that HAT can be used as a plug-and-play module for existing networks and enhance them. We further demonstrate significantly faster and more accurate performance than competitive counterparts for images with high resolution. Code is available at https://github.com/NVlabs/FasterViT.