Which Tokens to Use? Investigating Token Reduction in Vision Transformers
This work provides insights into token reduction efficiency for researchers and practitioners in computer vision, but it is incremental as it analyzes existing methods without introducing new ones.
The study investigated token reduction patterns across 10 methods and 4 image classification datasets in Vision Transformers, finding that Top-K pruning is a strong baseline and that pattern similarity moderately correlates with model performance.
Since the introduction of the Vision Transformer (ViT), researchers have sought to make ViTs more efficient by removing redundant information in the processed tokens. While different methods have been explored to achieve this goal, we still lack understanding of the resulting reduction patterns and how those patterns differ across token reduction methods and datasets. To close this gap, we set out to understand the reduction patterns of 10 different token reduction methods using four image classification datasets. By systematically comparing these methods on the different classification tasks, we find that the Top-K pruning method is a surprisingly strong baseline. Through in-depth analysis of the different methods, we determine that: the reduction patterns are generally not consistent when varying the capacity of the backbone model, the reduction patterns of pruning-based methods significantly differ from fixed radial patterns, and the reduction patterns of pruning-based methods are correlated across classification datasets. Finally we report that the similarity of reduction patterns is a moderate-to-strong proxy for model performance. Project page at https://vap.aau.dk/tokens.