Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation
For researchers in referring image segmentation, this work provides a novel zero-shot approach that leverages implicit grounding in image editing models, outperforming prior zero-shot methods.
The paper reveals that instruction-based image editing models exhibit strong foreground-background separability in their internal representations at the earliest denoising timestep, and proposes a training-free framework that repurposes these models for zero-shot referring image segmentation, achieving superior performance over existing zero-shot baselines on RefCOCO, RefCOCO+, and RefCOCOg.
Instruction-based image editing (IIE) models have recently demonstrated strong capability in modifying specific image regions according to natural language instructions, which implicitly requires identifying where an edit should be applied. This indicates that such models inherently perform language-conditioned visual semantic grounding. In this work, we investigate whether this implicit grounding can be leveraged for zero-shot referring image segmentation (RIS), a task that requires pixel-level localization of objects described by natural language expressions. Through systematic analysis, we reveal that strong foreground-background separability emerges in the internal representations of these models at the earliest denoising timestep, well before any visible image transformation occurs. Building on this insight, we propose a training-free framework that repurposes pretrained image editing models for RIS by exploiting their intermediate representations. Our approach decomposes localization into two complementary components: attention-based spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment. By leveraging feature-space separability, the framework produces accurate segmentation masks using only a single denoising step, without requiring full image synthesis. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that our method achieves superior performance over existing zero-shot baselines.