SOS: Segment Object System for Open-World Instance Segmentation With Object Priors
This work addresses the challenge of segmenting arbitrary unknown objects in images for computer vision applications, representing an incremental advance in open-world instance segmentation.
The paper tackles the problem of open-world instance segmentation by improving generalization and reducing background detections, achieving up to 81.6% precision improvement over state-of-the-art methods on datasets like COCO, LVIS, and ADE20k.
We propose an approach for Open-World Instance Segmentation (OWIS), a task that aims to segment arbitrary unknown objects in images by generalizing from a limited set of annotated object classes during training. Our Segment Object System (SOS) explicitly addresses the generalization ability and the low precision of state-of-the-art systems, which often generate background detections. To this end, we generate high-quality pseudo annotations based on the foundation model SAM. We thoroughly study various object priors to generate prompts for SAM, explicitly focusing the foundation model on objects. The strongest object priors were obtained by self-attention maps from self-supervised Vision Transformers, which we utilize for prompting SAM. Finally, the post-processed segments from SAM are used as pseudo annotations to train a standard instance segmentation system. Our approach shows strong generalization capabilities on COCO, LVIS, and ADE20k datasets and improves on the precision by up to 81.6% compared to the state-of-the-art. Source code is available at: https://github.com/chwilms/SOS