SaiT: Sparse Vision Transformers through Adaptive Token Pruning
This work addresses efficiency bottlenecks for vision transformer applications, offering a flexible tradeoff between accuracy and speed, but it is incremental as it builds on existing token pruning methods.
The authors tackled the problem of accelerating vision transformers by proposing a dense/sparse training framework with adaptive token pruning, which reduces FLOPs by 39%-43% and increases throughput by 67%-91% with less than 0.5% accuracy loss.
While vision transformers have achieved impressive results, effectively and efficiently accelerating these models can further boost performances. In this work, we propose a dense/sparse training framework to obtain a unified model, enabling weight sharing across various token densities. Thus one model offers a range of accuracy and throughput tradeoffs for different applications. Besides, we introduce adaptive token pruning to optimize the patch token sparsity based on the input image. In addition, we investigate knowledge distillation to enhance token selection capability in early transformer modules. Sparse adaptive image Transformer (SaiT) offers varying levels of model acceleration by merely changing the token sparsity on the fly. Specifically, SaiT reduces the computation complexity (FLOPs) by 39% - 43% and increases the throughput by 67% - 91% with less than 0.5% accuracy loss for various vision transformer models. Meanwhile, the same model also provides the zero accuracy drop option by skipping the sparsification step. SaiT achieves better accuracy and computation tradeoffs than state-of-the-art transformer and convolutional models.