SketchFFusion: Sketch-guided image editing with diffusion model
This work addresses the challenge of local fine-tuning in image editing for users, though it appears incremental as it builds on existing diffusion models with sketch-based conditioning.
The paper tackles the problem of sketch-guided image editing by proposing a sketch generation scheme that preserves image contours and adheres to user sketch styles, and a conditional diffusion model (SketchFFusion) based on sketch structure vectors to address issues like distortion and loss of fine details, demonstrating that it outperforms existing methods.
Sketch-guided image editing aims to achieve local fine-tuning of the image based on the sketch information provided by the user, while maintaining the original status of the unedited areas. Due to the high cost of acquiring human sketches, previous works mostly relied on edge maps as a substitute for sketches, but sketches possess more rich structural information. In this paper, we propose a sketch generation scheme that can preserve the main contours of an image and closely adhere to the actual sketch style drawn by the user. Simultaneously, current image editing methods often face challenges such as image distortion, training cost, and loss of fine details in the sketch. To address these limitations, We propose a conditional diffusion model (SketchFFusion) based on the sketch structure vector. We evaluate the generative performance of our model and demonstrate that it outperforms existing methods.