A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmark
This work addresses the challenge of generalizable 3D medical image segmentation for clinical applications, representing a novel method for a known bottleneck rather than an incremental improvement.
The authors tackled the problem of existing vision Transformers struggling with limited medical data and poor generalization across diverse medical image segmentation tasks by introducing MedFormer, a data-scalable Transformer that outperformed CNNs and vision Transformers on seven public datasets covering multiple modalities and medical targets.
Transformers have demonstrated remarkable performance in natural language processing and computer vision. However, existing vision Transformers struggle to learn from limited medical data and are unable to generalize on diverse medical image tasks. To tackle these challenges, we present MedFormer, a data-scalable Transformer designed for generalizable 3D medical image segmentation. Our approach incorporates three key elements: a desirable inductive bias, hierarchical modeling with linear-complexity attention, and multi-scale feature fusion that integrates spatial and semantic information globally. MedFormer can learn across tiny- to large-scale data without pre-training. Comprehensive experiments demonstrate MedFormer's potential as a versatile segmentation backbone, outperforming CNNs and vision Transformers on seven public datasets covering multiple modalities (e.g., CT and MRI) and various medical targets (e.g., healthy organs, diseased tissues, and tumors). We provide public access to our models and evaluation pipeline, offering solid baselines and unbiased comparisons to advance a wide range of downstream clinical applications.