Guided Image Synthesis via Initial Image Editing in Diffusion Model
This provides a novel method for precise image editing and synthesis tasks, such as repainting, by leveraging initial noise manipulation, which is incremental but offers new control mechanisms.
The paper tackles the problem of controlling image generation in diffusion models by manipulating the initial noise rather than the denoising process, showing that modifying pixel blocks in the initial latent image can influence specific regions of the generated image and achieve state-of-the-art performance in layout-to-image generation.
Diffusion models have the ability to generate high quality images by denoising pure Gaussian noise images. While previous research has primarily focused on improving the control of image generation through adjusting the denoising process, we propose a novel direction of manipulating the initial noise to control the generated image. Through experiments on stable diffusion, we show that blocks of pixels in the initial latent images have a preference for generating specific content, and that modifying these blocks can significantly influence the generated image. In particular, we show that modifying a part of the initial image affects the corresponding region of the generated image while leaving other regions unaffected, which is useful for repainting tasks. Furthermore, we find that the generation preferences of pixel blocks are primarily determined by their values, rather than their position. By moving pixel blocks with a tendency to generate user-desired content to user-specified regions, our approach achieves state-of-the-art performance in layout-to-image generation. Our results highlight the flexibility and power of initial image manipulation in controlling the generated image. Project Page: https://ut-mao.github.io/swap.github.io/