RegCL: Continual Adaptation of Segment Anything Model via Model Merging
This addresses the scalability issue of SAM for multi-domain segmentation tasks, though it is incremental as it builds on existing adapter-based adaptation methods.
The paper tackles the problem of catastrophic forgetting when adapting the Segment Anything Model (SAM) to multiple domains, proposing RegCL, a continual learning framework that merges domain-specific adaptation modules via weight optimization to maintain performance across domains without increasing model size or storing historical data.
To address the performance limitations of the Segment Anything Model (SAM) in specific domains, existing works primarily adopt adapter-based one-step adaptation paradigms. However, some of these methods are specific developed for specific domains. If used on other domains may lead to performance degradation. This issue of catastrophic forgetting severely limits the model's scalability. To address this issue, this paper proposes RegCL, a novel non-replay continual learning (CL) framework designed for efficient multi-domain knowledge integration through model merging. Specifically, RegCL incorporates the model merging algorithm into the continual learning paradigm by merging the parameters of SAM's adaptation modules (e.g., LoRA modules) trained on different domains. The merging process is guided by weight optimization, which minimizes prediction discrepancies between the merged model and each of the domain-specific models. RegCL effectively consolidates multi-domain knowledge while maintaining parameter efficiency, i.e., the model size remains constant regardless of the number of tasks, and no historical data storage is required. Experimental results demonstrate that RegCL achieves favorable continual learning performance across multiple downstream datasets, validating its effectiveness in dynamic scenarios.