Rethinking Referring Object Removal
This work addresses a domain-specific challenge in image editing for users needing precise object removal based on textual input, representing an incremental advancement in combining segmentation and inpainting tasks.
The paper tackles the problem of removing objects from images based on natural language descriptions and filling the resulting gaps with plausible content, achieving significant performance improvements over existing methods like diffusion models and two-stage approaches.
Referring object removal refers to removing the specific object in an image referred by natural language expressions and filling the missing region with reasonable semantics. To address this task, we construct the ComCOCO, a synthetic dataset consisting of 136,495 referring expressions for 34,615 objects in 23,951 image pairs. Each pair contains an image with referring expressions and the ground truth after elimination. We further propose an end-to-end syntax-aware hybrid mapping network with an encoding-decoding structure. Linguistic features are hierarchically extracted at the syntactic level and fused in the downsampling process of visual features with multi-head attention. The feature-aligned pyramid network is leveraged to generate segmentation masks and replace internal pixels with region affinity learned from external semantics in high-level feature maps. Extensive experiments demonstrate that our model outperforms diffusion models and two-stage methods which process the segmentation and inpainting task separately by a significant margin.