Explore In-Context Segmentation via Latent Diffusion Models
This work addresses in-context segmentation for computer vision, offering incremental improvements by adapting generative models to segmentation tasks.
The paper tackled in-context segmentation by leveraging latent diffusion models, proposing a two-stage masking strategy and augmented pseudo-masking target to improve performance, and achieved comparable or stronger results than previous models on a new benchmark covering image and video datasets.
In-context segmentation has drawn increasing attention with the advent of vision foundation models. Its goal is to segment objects using given reference images. Most existing approaches adopt metric learning or masked image modeling to build the correlation between visual prompts and input image queries. This work approaches the problem from a fresh perspective - unlocking the capability of the latent diffusion model (LDM) for in-context segmentation and investigating different design choices. Specifically, we examine the problem from three angles: instruction extraction, output alignment, and meta-architectures. We design a two-stage masking strategy to prevent interfering information from leaking into the instructions. In addition, we propose an augmented pseudo-masking target to ensure the model predicts without forgetting the original images. Moreover, we build a new and fair in-context segmentation benchmark that covers both image and video datasets. Experiments validate the effectiveness of our approach, demonstrating comparable or even stronger results than previous specialist or visual foundation models. We hope our work inspires others to rethink the unification of segmentation and generation.