FilterViT and DropoutViT
This work addresses computational bottlenecks in ViTs for computer vision tasks, offering a more efficient and interpretable approach, though it appears incremental as it builds on existing ViT architectures.
The paper tackles the computational inefficiency of Vision Transformers (ViTs) when performing attention on high-resolution feature maps by introducing a filter attention mechanism that selects top-K informative pixels, reducing tokens and complexity. Experimental results show improved parameter efficiency, computational speed, and accuracy compared to existing models.
In this study, we introduce an enhanced version of ViT that conducts attention-based QKV operations during the initial stages of downsampling. Performing attention directly on high-resolution feature maps is computationally demanding due to the large size and numerous tokens. To mitigate this, we propose a filter attention mechanism that uses a Filter Block to create a salient mask (Filter Mask) for selecting the most informative pixels for attention. The Filter Block scores the pixels of the feature map, and we sort these scores to retain only the top K pixels (with K varying across layers). This approach effectively decreases the number of tokens involved in the attention computation, reducing computational complexity and boosting processing speed. Furthermore, the salient mask provides interpretability, as the model focuses on regions of the image most critical to the outcome. Our experimental results show that this model improves parameter efficiency and computational speed while enhancing accuracy. Compared to existing models, our approach significantly reduces resource consumption while maintaining high performance.