SwInception -- Local Attention Meets Convolutions
For medical image segmentation, this work incrementally improves sparse vision transformers by combining local attention with multi-scale convolutions to address overfitting.
SwInception enhances Swin transformers by integrating Inception blocks into feed-forward layers to improve inductive bias and reduce overfitting on small medical datasets, achieving state-of-the-art performance on eleven medical segmentation datasets including the Medical Segmentation Decathlon and Beyond the Cranial Vault.
Sparse vision transformers have gained popularity as efficient encoders for medical volumetric segmentation, with Swin emerging as a prominent choice. Swin uses local attention to reduce complexity and yields excellent performance for many tasks but still tends to overfit on small datasets. To mitigate this weakness, we propose a novel architecture that further enhances Swin's inductive bias by introducing Inception blocks in the feed-forward layers. The introduction of these multi-branch convolutions enables more direct reasoning over local, multi-scale features within the transformer block. We have also modified the decoder layers in order to capture finer details using fewer parameters. We demonstrate a performance improvement on eleven different medical datasets through extensive experimentation. We specifically showcase advancements over the previous state-of-the-art backbones on benchmark challenges like the Medical Segmentation Decathlon and Beyond the Cranial Vault. By showing that the existing inductive bias in Swin can be further improved, our work presents a promising avenue for enhancing the capabilities of sparse vision transformers for both medical and natural image segmentation tasks. Code and pre-trained weights can be accessed at https://github.com/Eiphodos/SwInception.