AFIDAF: Alternating Fourier and Image Domain Adaptive Filters as an Efficient Alternative to Attention in ViTs
This provides an efficient alternative to attention for building vision backbones, potentially benefiting applications requiring lightweight models in computer vision.
The paper tackled the computational inefficiency of attention in vision transformers by proposing an alternating Fourier and image domain filtering approach, achieving state-of-the-art performance among lightweight models on ImageNet-1K classification and improving downstream tasks like object detection and segmentation.
We propose and demonstrate an alternating Fourier and image domain filtering approach for feature extraction as an efficient alternative to build a vision backbone without using the computationally intensive attention. The performance among the lightweight models reaches the state-of-the-art level on ImageNet-1K classification, and improves downstream tasks on object detection and segmentation consistently as well. Our approach also serves as a new tool to compress vision transformers (ViTs).