SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation
This work addresses the challenge of reducing labeling and training costs for segmentation tasks, with potential applications in video object and open-vocabulary segmentation, though it appears incremental as it builds on existing vision foundation models.
The paper tackles the problem of in-context segmentation, which segments novel images using a few labeled examples, by proposing SEGIC, an end-to-end framework that leverages a vision foundation model to capture dense relationships and achieves state-of-the-art performance on one-shot segmentation benchmarks.
In-context segmentation aims at segmenting novel images using a few labeled example images, termed as "in-context examples", exploring content similarities between examples and the target. The resulting models can be generalized seamlessly to novel segmentation tasks, significantly reducing the labeling and training costs compared with conventional pipelines. However, in-context segmentation is more challenging than classic ones requiring the model to learn segmentation rules conditioned on a few samples. Unlike previous work with ad-hoc or non-end-to-end designs, we propose SEGIC, an end-to-end segment-in-context framework built upon a single vision foundation model (VFM). In particular, SEGIC leverages the emergent correspondence within VFM to capture dense relationships between target images and in-context samples. As such, information from in-context samples is then extracted into three types of instructions, i.e. geometric, visual, and meta instructions, serving as explicit conditions for the final mask prediction. SEGIC is a straightforward yet effective approach that yields state-of-the-art performance on one-shot segmentation benchmarks. Notably, SEGIC can be easily generalized to diverse tasks, including video object segmentation and open-vocabulary segmentation. Code will be available at https://github.com/MengLcool/SEGIC.