CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Models
This addresses the issue of unintended hallucinations in object removal for image editing applications, offering a flexible solution for users, though it is incremental as it builds on existing diffusion-based inpainting techniques.
The paper tackles the problem of object removal in image inpainting using diffusion models, which often hallucinate removed objects, by introducing CLIPAway, a method that uses CLIP embeddings to focus on background regions, resulting in enhanced inpainting accuracy and quality without requiring specialized training or manual annotations.
Advanced image editing techniques, particularly inpainting, are essential for seamlessly removing unwanted elements while preserving visual integrity. Traditional GAN-based methods have achieved notable success, but recent advancements in diffusion models have produced superior results due to their training on large-scale datasets, enabling the generation of remarkably realistic inpainted images. Despite their strengths, diffusion models often struggle with object removal tasks without explicit guidance, leading to unintended hallucinations of the removed object. To address this issue, we introduce CLIPAway, a novel approach leveraging CLIP embeddings to focus on background regions while excluding foreground elements. CLIPAway enhances inpainting accuracy and quality by identifying embeddings that prioritize the background, thus achieving seamless object removal. Unlike other methods that rely on specialized training datasets or costly manual annotations, CLIPAway provides a flexible, plug-and-play solution compatible with various diffusion-based inpainting techniques.