ENACT: Entropy-based Clustering of Attention Input for Reducing the Computational Needs of Object Detection Transformers
This work addresses the computational bottleneck for researchers and practitioners using transformers in object detection, though it is incremental as it builds on existing transformer methods.
The authors tackled the high computational demands of object detection transformers by clustering attention inputs based on entropy to reduce GPU memory usage, achieving a slight degradation in accuracy on the COCO dataset.
Transformers demonstrate competitive performance in terms of precision on the problem of vision-based object detection. However, they require considerable computational resources due to the quadratic size of the attention weights. In this work, we propose to cluster the transformer input on the basis of its entropy, due to its similarity between same object pixels. This is expected to reduce GPU usage during training, while maintaining reasonable accuracy. This idea is realized with an implemented module that is called ENtropy-based Attention Clustering for detection Transformers (ENACT), which serves as a plug-in to any multi-head self-attention based transformer network. Experiments on the COCO object detection dataset and three detection transformers demonstrate that the requirements on memory are reduced, while the detection accuracy is degraded only slightly. The code of the ENACT module is available at https://github.com/GSavathrakis/ENACT.