Paint by Inpaint: Learning to Add Image Objects by Removing Them First
This addresses the problem of seamless image editing for users by automating object addition without manual input, though it is incremental as it builds on existing diffusion and inpainting techniques.
The paper tackles the challenge of adding objects to images based on textual instructions without user-provided masks by training a diffusion model to invert the inpainting process, using a curated dataset of image pairs with object-removed versions, and shows that the model surpasses existing models in object addition and general editing tasks.
Image editing has advanced significantly with the introduction of text-conditioned diffusion models. Despite this progress, seamlessly adding objects to images based on textual instructions without requiring user-provided input masks remains a challenge. We address this by leveraging the insight that removing objects (Inpaint) is significantly simpler than its inverse process of adding them (Paint), attributed to inpainting models that benefit from segmentation mask guidance. Capitalizing on this realization, by implementing an automated and extensive pipeline, we curate a filtered large-scale image dataset containing pairs of images and their corresponding object-removed versions. Using these pairs, we train a diffusion model to inverse the inpainting process, effectively adding objects into images. Unlike other editing datasets, ours features natural target images instead of synthetic ones while ensuring source-target consistency by construction. Additionally, we utilize a large Vision-Language Model to provide detailed descriptions of the removed objects and a Large Language Model to convert these descriptions into diverse, natural-language instructions. Our quantitative and qualitative results show that the trained model surpasses existing models in both object addition and general editing tasks. Visit our project page for the released dataset and trained models at https://rotsteinnoam.github.io/Paint-by-Inpaint.