Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers
This work addresses the challenge of model selection for practitioners and researchers by providing a standardized benchmark for efficiency-oriented transformers, though it is incremental as it focuses on comparative analysis rather than introducing new methods.
The authors tackled the problem of fair efficiency comparison among vision transformers by conducting a large-scale benchmark of over 45 models for image classification, evaluating accuracy, speed, and memory usage, and found that ViT remains Pareto optimal and hybrid attention-CNN models are memory- and parameter-efficient.
Self-attention in Transformers comes with a high computational cost because of their quadratic computational complexity, but their effectiveness in addressing problems in language and vision has sparked extensive research aimed at enhancing their efficiency. However, diverse experimental conditions, spanning multiple input domains, prevent a fair comparison based solely on reported results, posing challenges for model selection. To address this gap in comparability, we perform a large-scale benchmark of more than 45 models for image classification, evaluating key efficiency aspects, including accuracy, speed, and memory usage. Our benchmark provides a standardized baseline for efficiency-oriented transformers. We analyze the results based on the Pareto front -- the boundary of optimal models. Surprisingly, despite claims of other models being more efficient, ViT remains Pareto optimal across multiple metrics. We observe that hybrid attention-CNN models exhibit remarkable inference memory- and parameter-efficiency. Moreover, our benchmark shows that using a larger model in general is more efficient than using higher resolution images. Thanks to our holistic evaluation, we provide a centralized resource for practitioners and researchers, facilitating informed decisions when selecting or developing efficient transformers.