Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning
This addresses the challenge of distribution shift in segmentation foundation models for applications like medical imaging and camouflage detection, though it is incremental as it builds on existing models.
The paper tackles the problem of foundation models like Segment Anything performing poorly on out-of-distribution data, such as camouflaged and medical images, by introducing SlotSAM, which uses object-centric learning to enhance generalization, achieving significant improvements in performance metrics.
Foundation models have made incredible strides in achieving zero-shot or few-shot generalization, leveraging prompt engineering to mimic the problem-solving approach of human intelligence. However, when it comes to some foundation models like Segment Anything, there is still a challenge in performing well on out-of-distribution data, including camouflaged and medical images. Inconsistent prompting strategies during fine-tuning and testing further compound the issue, leading to decreased performance. Drawing inspiration from how human cognition processes new environments, we introduce SlotSAM, a method that reconstructs features from the encoder in a self-supervised manner to create object-centric representations. These representations are then integrated into the foundation model, bolstering its object-level perceptual capabilities while reducing the impact of distribution-related variables. The beauty of SlotSAM lies in its simplicity and adaptability to various tasks, making it a versatile solution that significantly enhances the generalization abilities of foundation models. Through limited parameter fine-tuning in a bootstrap manner, our approach paves the way for improved generalization in novel environments. The code is available at github.com/lytang63/SlotSAM.