Training-Free Layout Control with Cross-Attention Guidance
This addresses the issue of precise spatial control in text-to-image generation for users needing specific compositions, representing an incremental improvement over existing methods.
The paper tackles the problem of diffusion-based image generators ignoring textual instructions for spatial layout, proposing a training-free method that manipulates cross-attention layers to achieve robust layout control, with backward guidance outperforming forward guidance and prior work in evaluations on three benchmarks.
Recent diffusion-based generators can produce high-quality images from textual prompts. However, they often disregard textual instructions that specify the spatial layout of the composition. We propose a simple approach that achieves robust layout control without the need for training or fine-tuning of the image generator. Our technique manipulates the cross-attention layers that the model uses to interface textual and visual information and steers the generation in the desired direction given, e.g., a user-specified layout. To determine how to best guide attention, we study the role of attention maps and explore two alternative strategies, forward and backward guidance. We thoroughly evaluate our approach on three benchmarks and provide several qualitative examples and a comparative analysis of the two strategies that demonstrate the superiority of backward guidance compared to forward guidance, as well as prior work. We further demonstrate the versatility of layout guidance by extending it to applications such as editing the layout and context of real images.