Learning to Merge Tokens in Vision Transformers
This addresses efficiency for deploying large-scale vision transformers in real-world systems, but it is incremental as it builds on existing token reduction methods.
The paper tackles the high computational cost of scaling vision transformers by introducing PatchMerger, a module that merges tokens between layers to reduce processing, achieving significant speedup while matching original performance.
Transformers are widely applied to solve natural language understanding and computer vision tasks. While scaling up these architectures leads to improved performance, it often comes at the expense of much higher computational costs. In order for large-scale models to remain practical in real-world systems, there is a need for reducing their computational overhead. In this work, we present the PatchMerger, a simple module that reduces the number of patches or tokens the network has to process by merging them between two consecutive intermediate layers. We show that the PatchMerger achieves a significant speedup across various model sizes while matching the original performance both upstream and downstream after fine-tuning.