MRL: Learning to Mix with Attention and Convolutions
This work addresses the need for efficient and effective feature mixing in neural networks for vision applications, particularly in medical imaging, though it appears incremental as it hybridizes existing methods.
The paper tackles the problem of mixing input features in vision tasks by proposing a new architectural block, MRL, which combines self-attention for regional-scale mixing and convolutions for local-scale mixing, resulting in improved performance across classification, object detection, and segmentation, with DICE scores of 0.843, 0.855, and 0.892 on histology datasets.
In this paper, we present a new neural architectural block for the vision domain, named Mixing Regionally and Locally (MRL), developed with the aim of effectively and efficiently mixing the provided input features. We bifurcate the input feature mixing task as mixing at a regional and local scale. To achieve an efficient mix, we exploit the domain-wide receptive field provided by self-attention for regional-scale mixing and convolutional kernels restricted to local scale for local-scale mixing. More specifically, our proposed method mixes regional features associated with local features within a defined region, followed by a local-scale features mix augmented by regional features. Experiments show that this hybridization of self-attention and convolution brings improved capacity, generalization (right inductive bias), and efficiency. Under similar network settings, MRL outperforms or is at par with its counterparts in classification, object detection, and segmentation tasks. We also show that our MRL-based network architecture achieves state-of-the-art performance for H&E histology datasets. We achieved DICE of 0.843, 0.855, and 0.892 for Kumar, CoNSep, and CPM-17 datasets, respectively, while highlighting the versatility offered by the MRL framework by incorporating layers like group convolutions to improve dataset-specific generalization.