FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior
This addresses the need for flexible image composition tools for users in computer vision, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of generic image composition by integrating multiple input images into a coherent output, using a diffusion prior to optimize composed images towards high-density regions, achieving superior zero-shot performance validated through extensive experiments.
We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. Rather than concentrating on specific use cases such as appearance editing (image harmonization) or semantic editing (semantic image composition), we showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition applicable to both scenarios. We observe that the pre-trained diffusion models automatically identify simple copy-paste boundary areas as low-density regions during denoising. Building on this insight, we propose to optimize the composed image towards high-density regions guided by the diffusion prior. In addition, we introduce a novel maskguided loss to further enable flexible semantic image composition. Extensive experiments validate the superiority of our approach in achieving generic zero-shot image composition. Additionally, our approach shows promising potential in various tasks, such as object removal and multiconcept customization.