Spatial-Aware Latent Initialization for Controllable Image Generation
This addresses a key limitation in controllable image generation for users needing precise spatial control, though it is incremental as it builds on existing layout guidance frameworks.
The paper tackles the problem of text-to-image diffusion models struggling to adhere to spatial layout instructions by proposing a spatial-aware initialization noise method, which significantly improves layout guidance effectiveness while preserving image quality, as evaluated on Stable Diffusion and COCO datasets.
Recently, text-to-image diffusion models have demonstrated impressive ability to generate high-quality images conditioned on the textual input. However, these models struggle to accurately adhere to textual instructions regarding spatial layout information. While previous research has primarily focused on aligning cross-attention maps with layout conditions, they overlook the impact of the initialization noise on the layout guidance. To achieve better layout control, we propose leveraging a spatial-aware initialization noise during the denoising process. Specifically, we find that the inverted reference image with finite inversion steps contains valuable spatial awareness regarding the object's position, resulting in similar layouts in the generated images. Based on this observation, we develop an open-vocabulary framework to customize a spatial-aware initialization noise for each layout condition. Without modifying other modules except the initialization noise, our approach can be seamlessly integrated as a plug-and-play module within other training-free layout guidance frameworks. We evaluate our approach quantitatively and qualitatively on the available Stable Diffusion model and COCO dataset. Equipped with the spatial-aware latent initialization, our method significantly improves the effectiveness of layout guidance while preserving high-quality content.