pFedSAM: Personalized Federated Learning of Segment Anything Model for Medical Image Segmentation
This work addresses privacy-preserving medical image segmentation for institutions with heterogeneous data, representing an incremental advancement in federated learning methods.
The paper tackled the challenge of applying the Segment Anything Model (SAM) in federated learning for medical image segmentation by proposing pFedSAM, a personalized framework that integrates a localized mixture-of-experts and a teacher-student distillation mechanism, resulting in improved segmentation performance, robust cross-domain adaptation, and reduced communication overhead.
Medical image segmentation is crucial for computer-aided diagnosis, yet privacy constraints hinder data sharing across institutions. Federated learning addresses this limitation, but existing approaches often rely on lightweight architectures that struggle with complex, heterogeneous data. Recently, the Segment Anything Model (SAM) has shown outstanding segmentation capabilities; however, its massive encoder poses significant challenges in federated settings. In this work, we present the first personalized federated SAM framework tailored for heterogeneous data scenarios in medical image segmentation. Our framework integrates two key innovations: (1) a personalized strategy that aggregates only the global parameters to capture cross-client commonalities while retaining the designed L-MoE (Localized Mixture-of-Experts) component to preserve domain-specific features; and (2) a decoupled global-local fine-tuning mechanism that leverages a teacher-student paradigm via knowledge distillation to bridge the gap between the global shared model and the personalized local models, thereby mitigating overgeneralization. Extensive experiments on two public datasets validate that our approach significantly improves segmentation performance, achieves robust cross-domain adaptation, and reduces communication overhead.