Interpreting and Improving Attention From the Perspective of Large Kernel Convolution
This work addresses challenges in visual models for resource-constrained and data-limited applications, offering a practical solution, though it appears incremental by combining existing concepts.
The paper tackled the problem of attention mechanisms being computationally expensive and lacking spatial inductive biases for local features in data-scarce scenarios by introducing Large Kernel Convolutional Attention (LKCA), which achieved competitive performance in image classification on datasets like CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet, outperforming conventional attention and vision transformers in compact models.
Attention mechanisms have significantly advanced visual models by capturing global context effectively. However, their reliance on large-scale datasets and substantial computational resources poses challenges in data-scarce and resource-constrained scenarios. Moreover, traditional self-attention mechanisms lack inherent spatial inductive biases, making them suboptimal for modeling local features critical to tasks involving smaller datasets. In this work, we introduce Large Kernel Convolutional Attention (LKCA), a novel formulation that reinterprets attention operations as a single large-kernel convolution. This design unifies the strengths of convolutional architectures locality and translation invariance with the global context modeling capabilities of self-attention. By embedding these properties into a computationally efficient framework, LKCA addresses key limitations of traditional attention mechanisms. The proposed LKCA achieves competitive performance across various visual tasks, particularly in data-constrained settings. Experimental results on CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet demonstrate its ability to excel in image classification, outperforming conventional attention mechanisms and vision transformers in compact model settings. These findings highlight the effectiveness of LKCA in bridging local and global feature modeling, offering a practical and robust solution for real-world applications with limited data and resources.