Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models
This addresses the challenge of personalized content editing for users needing precise object integration in images, representing an incremental improvement over existing methods.
The paper tackles the problem of achieving seamless and identity-consistent personalized image editing with pretrained diffusion models, proposing a training-free optimization framework that uses energy guidance and a coarse-to-fine strategy, resulting in high-quality object replacement even with large domain gaps.
The rapid advancement of pretrained text-driven diffusion models has significantly enriched applications in image generation and editing. However, as the demand for personalized content editing increases, new challenges emerge especially when dealing with arbitrary objects and complex scenes. Existing methods usually mistakes mask as the object shape prior, which struggle to achieve a seamless integration result. The mostly used inversion noise initialization also hinders the identity consistency towards the target object. To address these challenges, we propose a novel training-free framework that formulates personalized content editing as the optimization of edited images in the latent space, using diffusion models as the energy function guidance conditioned by reference text-image pairs. A coarse-to-fine strategy is proposed that employs text energy guidance at the early stage to achieve a natural transition toward the target class and uses point-to-point feature-level image energy guidance to perform fine-grained appearance alignment with the target object. Additionally, we introduce the latent space content composition to enhance overall identity consistency with the target. Extensive experiments demonstrate that our method excels in object replacement even with a large domain gap, highlighting its potential for high-quality, personalized image editing.