Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models
This work addresses the problem of segmenting images based on text descriptions without paired training data for researchers and practitioners in computer vision, representing an incremental advancement by integrating generative models into a task previously dominated by discriminative approaches.
The paper tackles zero-shot referring image segmentation by leveraging generative models like Stable Diffusion to understand relationships between visual elements and text, achieving comparable performance to existing SOTA weakly-supervised models without a proposal generator and outperforming them significantly when combined with discriminative models.
Zero-shot referring image segmentation is a challenging task because it aims to find an instance segmentation mask based on the given referring descriptions, without training on this type of paired data. Current zero-shot methods mainly focus on using pre-trained discriminative models (e.g., CLIP). However, we have observed that generative models (e.g., Stable Diffusion) have potentially understood the relationships between various visual elements and text descriptions, which are rarely investigated in this task. In this work, we introduce a novel Referring Diffusional segmentor (Ref-Diff) for this task, which leverages the fine-grained multi-modal information from generative models. We demonstrate that without a proposal generator, a generative model alone can achieve comparable performance to existing SOTA weakly-supervised models. When we combine both generative and discriminative models, our Ref-Diff outperforms these competing methods by a significant margin. This indicates that generative models are also beneficial for this task and can complement discriminative models for better referring segmentation. Our code is publicly available at https://github.com/kodenii/Ref-Diff.