Utilizing dynamic sparsity on pretrained DETR
This work addresses computational efficiency for deploying pretrained vision transformers, offering a practical, input-adaptive method that is incremental over existing sparsification techniques.
The paper tackles the challenge of efficient inference in transformer-based vision models like DETR by exploiting sparsity in MLP layers, introducing Micro-Gated Sparsification (MGS) which achieves 85-95% activation sparsity and maintains or improves performance on COCO.
Efficient inference with transformer-based models remains a challenge, especially in vision tasks like object detection. We analyze the inherent sparsity in the MLP layers of DETR and introduce two methods to exploit it without retraining. First, we propose Static Indicator-Based Sparsification (SIBS), a heuristic method that predicts neuron inactivity based on fixed activation patterns. While simple, SIBS offers limited gains due to the input-dependent nature of sparsity. To address this, we introduce Micro-Gated Sparsification (MGS), a lightweight gating mechanism trained on top of a pretrained DETR. MGS predicts dynamic sparsity using a small linear layer and achieves up to 85 to 95% activation sparsity. Experiments on the COCO dataset show that MGS maintains or even improves performance while significantly reducing computation. Our method offers a practical, input-adaptive approach to sparsification, enabling efficient deployment of pretrained vision transformers without full model retraining.