Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
This addresses the challenge of deploying large foundation models like SAM in privacy-sensitive medical settings with incremental improvements in efficiency and adaptation.
The paper tackles the problem of adapting the Segment Anything Model (SAM) for continual learning in medical image segmentation, where joint training is infeasible due to privacy constraints, by introducing a lightweight Alignment Layer and a continual learning strategy (CA-SAM) that achieves state-of-the-art performance across nine datasets.
In medical image segmentation, heterogeneous privacy policies across institutions often make joint training on pooled datasets infeasible, motivating continual image segmentation-learning from data streams without catastrophic forgetting. While the Segment Anything Model (SAM) offers strong zero-shot priors and has been widely fine-tuned across downstream tasks, its large parameter count and computational overhead challenge practical deployment. This paper demonstrates that the SAM paradigm is highly promising once its computational efficiency and performance can be balanced. To this end, we introduce the Alignment Layer, a lightweight, plug-and-play module which aligns encoder-decoder feature distributions to efficiently adapt SAM to specific medical images, improving accuracy while reducing computation. Building on SAM and the Alignment Layer, we then propose Continual Alignment for SAM (CA-SAM), a continual learning strategy that automatically adapts the appropriate Alignment Layer to mitigate catastrophic forgetting, while leveraging SAM's zero-shot priors to preserve strong performance on unseen medical datasets. Experimented across nine medical segmentation datasets under continual-learning scenario, CA-SAM achieves state-of-the-art performance. Our code, models and datasets will be released on \mbox{https://github.com/azzzzyo/Continual-Alignment-for-SAM.}