Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations
This work addresses the scalability issue in instance segmentation for computer vision researchers and practitioners by reducing annotation costs, though it builds incrementally on existing open-vocabulary approaches.
The paper tackles the problem of open-vocabulary instance segmentation by eliminating the need for manual mask annotations, which are labor-intensive and limit scalability to new categories. The result is a method that uses pseudo-mask annotations from vision-language models, achieving significant improvements in mAP scores on MS-COCO and OpenImages datasets compared to state-of-the-art methods trained with manual masks.
Existing instance segmentation models learn task-specific information using manual mask annotations from base (training) categories. These mask annotations require tremendous human effort, limiting the scalability to annotate novel (new) categories. To alleviate this problem, Open-Vocabulary (OV) methods leverage large-scale image-caption pairs and vision-language models to learn novel categories. In summary, an OV method learns task-specific information using strong supervision from base annotations and novel category information using weak supervision from image-captions pairs. This difference between strong and weak supervision leads to overfitting on base categories, resulting in poor generalization towards novel categories. In this work, we overcome this issue by learning both base and novel categories from pseudo-mask annotations generated by the vision-language model in a weakly supervised manner using our proposed Mask-free OVIS pipeline. Our method automatically generates pseudo-mask annotations by leveraging the localization ability of a pre-trained vision-language model for objects present in image-caption pairs. The generated pseudo-mask annotations are then used to supervise an instance segmentation model, freeing the entire pipeline from any labour-expensive instance-level annotations and overfitting. Our extensive experiments show that our method trained with just pseudo-masks significantly improves the mAP scores on the MS-COCO dataset and OpenImages dataset compared to the recent state-of-the-art methods trained with manual masks. Codes and models are provided in https://vibashan.github.io/ovis-web/.