RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation
For medical image segmentation practitioners, this provides a practical method to achieve robustness against common image corruptions without sacrificing clean-domain performance.
RobustMedSAM improves medical image segmentation under realistic corruptions (noise, blur, motion artifacts) by fusing the image encoder from MedSAM and the mask decoder from RobustSAM, achieving a +0.106 Dice improvement (from 0.613 to 0.719) over SAM on degraded images.
Medical image segmentation models built on Segment Anything Model (SAM) achieve strong performance on clean benchmarks, yet their reliability often degrades under realistic image corruptions such as noise, blur, motion artifacts, and modality-specific distortions. Existing approaches address either medical-domain adaptation or corruption robustness, but not both jointly. In SAM, we find that these capabilities are concentrated in complementary modules: the image encoder preserves medical priors, while the mask decoder governs corruption robustness. Motivated by this observation, we propose RobustMedSAM, which adopts module-wise checkpoint fusion by initializing the image encoder from MedSAM and the mask decoder from RobustSAM under a shared ViT-B architecture. We then fine-tune only the mask decoder on 35 medical datasets from MedSegBench, spanning six imaging modalities and 12 corruption types, while freezing the remaining components to preserve pretrained medical representations. We additionally investigate an SVD-based parameter-efficient variant for limited encoder adaptation. Experiments on both in-distribution and out-of-distribution benchmarks show that RobustMedSAM improves degraded-image Dice from 0.613 to 0.719 (+0.106) over SAM, demonstrating that structured fusion of complementary pretrained models is an effective and practical approach for robust medical image segmentation.