InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Models
This addresses spatial alignment issues in text-to-image generation for users needing accurate object placement, representing a strong incremental improvement.
The paper tackles the problem of text-to-image diffusion models failing to capture spatial relations in prompts by introducing InfSplign, an inference-time method that adjusts noise using a compound loss, achieving state-of-the-art performance with substantial gains over existing baselines.
Text-to-image (T2I) diffusion models generate high-quality images but often fail to capture the spatial relations specified in text prompts. This limitation can be traced to two factors: lack of fine-grained spatial supervision in training data and inability of text embeddings to encode spatial semantics. We introduce InfSplign, a training-free inference-time method that improves spatial alignment by adjusting the noise through a compound loss in every denoising step. Proposed loss leverages different levels of cross-attention maps extracted from the backbone decoder to enforce accurate object placement and a balanced object presence during sampling. The method is lightweight, plug-and-play, and compatible with any diffusion backbone. Our comprehensive evaluations on VISOR and T2I-CompBench show that InfSplign establishes a new state-of-the-art (to the best of our knowledge), achieving substantial performance gains over the strongest existing inference-time baselines and even outperforming the fine-tuning-based methods. Codebase is available at GitHub.