Describe, Don't Dictate: Semantic Image Editing with Natural Language Intent
It addresses the problem of semantic image editing for users needing precise control over image modifications, though it is incremental as it builds on existing text-to-image models.
The paper tackles semantic image editing by reframing instruction-based editing as reference-image-based text-to-image generation, proposing DescriptiveEdit, which improves editing accuracy and consistency on the Emu Edit benchmark.
Despite the progress in text-to-image generation, semantic image editing remains a challenge. Inversion-based algorithms unavoidably introduce reconstruction errors, while instruction-based models mainly suffer from limited dataset quality and scale. To address these problems, we propose a descriptive-prompt-based editing framework, named DescriptiveEdit. The core idea is to re-frame `instruction-based image editing' as `reference-image-based text-to-image generation', which preserves the generative power of well-trained Text-to-Image models without architectural modifications or inversion. Specifically, taking the reference image and a prompt as input, we introduce a Cross-Attentive UNet, which newly adds attention bridges to inject reference image features into the prompt-to-edit-image generation process. Owing to its text-to-image nature, DescriptiveEdit overcomes limitations in instruction dataset quality, integrates seamlessly with ControlNet, IP-Adapter, and other extensions, and is more scalable. Experiments on the Emu Edit benchmark show it improves editing accuracy and consistency.