Reference-based Image Composition with Sketch via Structure-aware Diffusion Model
This work addresses the need for enhanced editability in image generation for users, though it is incremental as it builds on pre-trained diffusion models.
The paper tackles the problem of fine-grained image editing by introducing a multi-input-conditioned image composition model that uses sketches and reference images to edit or complete image sub-parts with desired structure and content, resulting in a method that enables user-driven modifications of arbitrary scenes.
Recent remarkable improvements in large-scale text-to-image generative models have shown promising results in generating high-fidelity images. To further enhance editability and enable fine-grained generation, we introduce a multi-input-conditioned image composition model that incorporates a sketch as a novel modal, alongside a reference image. Thanks to the edge-level controllability using sketches, our method enables a user to edit or complete an image sub-part with a desired structure (i.e., sketch) and content (i.e., reference image). Our framework fine-tunes a pre-trained diffusion model to complete missing regions using the reference image while maintaining sketch guidance. Albeit simple, this leads to wide opportunities to fulfill user needs for obtaining the in-demand images. Through extensive experiments, we demonstrate that our proposed method offers unique use cases for image manipulation, enabling user-driven modifications of arbitrary scenes.