Spectral-Adaptive Modulation Networks for Visual Perception
This work addresses the need for better theoretical insights and performance in visual perception models, though it appears incremental by building on existing spectral analysis and mixer concepts.
The paper tackled the problem of understanding and optimizing the spectral properties of 2D convolution and self-attention in vision models, resulting in SPANetV2, which outperforms state-of-the-art models on tasks like ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation.
Recent studies have shown that 2D convolution and self-attention exhibit distinct spectral behaviors, and optimizing their spectral properties can enhance vision model performance. However, theoretical analyses remain limited in explaining why 2D convolution is more effective in high-pass filtering than self-attention and why larger kernels favor shape bias, akin to self-attention. In this paper, we employ graph spectral analysis to theoretically simulate and compare the frequency responses of 2D convolution and self-attention within a unified framework. Our results corroborate previous empirical findings and reveal that node connectivity, modulated by window size, is a key factor in shaping spectral functions. Leveraging this insight, we introduce a \textit{spectral-adaptive modulation} (SPAM) mixer, which processes visual features in a spectral-adaptive manner using multi-scale convolutional kernels and a spectral re-scaling mechanism to refine spectral components. Based on SPAM, we develop SPANetV2 as a novel vision backbone. Extensive experiments demonstrate that SPANetV2 outperforms state-of-the-art models across multiple vision tasks, including ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation.