Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentation
This work addresses the challenge of costly manual annotation in referring image segmentation, offering a scalable solution for open-world applications.
The paper tackles the problem of generating high-quality pseudo-supervision for referring image segmentation (RIS) by automatically creating segmentation masks with distinctive referring expressions, eliminating the need for manual labeling. The result is a method that significantly outperforms state-of-the-art weakly and zero-shot approaches on RIS benchmarks and surpasses fully supervised methods in unseen domains.
We propose a new framework that automatically generates high-quality segmentation masks with their referring expressions as pseudo supervisions for referring image segmentation (RIS). These pseudo supervisions allow the training of any supervised RIS methods without the cost of manual labeling. To achieve this, we incorporate existing segmentation and image captioning foundation models, leveraging their broad generalization capabilities. However, the naive incorporation of these models may generate non-distinctive expressions that do not distinctively refer to the target masks. To address this challenge, we propose two-fold strategies that generate distinctive captions: 1) 'distinctive caption sampling', a new decoding method for the captioning model, to generate multiple expression candidates with detailed words focusing on the target. 2) 'distinctiveness-based text filtering' to further validate the candidates and filter out those with a low level of distinctiveness. These two strategies ensure that the generated text supervisions can distinguish the target from other objects, making them appropriate for the RIS annotations. Our method significantly outperforms both weakly and zero-shot SoTA methods on the RIS benchmark datasets. It also surpasses fully supervised methods in unseen domains, proving its capability to tackle the open-world challenge within RIS. Furthermore, integrating our method with human annotations yields further improvements, highlighting its potential in semi-supervised learning applications.