Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking
This addresses layout-to-image generation for applications requiring precise spatial control, but it is incremental as it builds on existing autoregressive methods.
The paper tackled the challenge of extending autoregressive models to layout-conditioned image generation by proposing SMARLI, a framework that uses structured masking and GRPO post-training to integrate layout constraints, achieving superior layout-aware control without compromising generation quality.
While autoregressive (AR) models have demonstrated remarkable success in image generation, extending them to layout-conditioned generation remains challenging due to the sparse nature of layout conditions and the risk of feature entanglement. We present Structured Masking for AR-based Layout-to-Image (SMARLI), a novel framework for layoutto-image generation that effectively integrates spatial layout constraints into AR-based image generation. To equip AR model with layout control, a specially designed structured masking strategy is applied to attention computation to govern the interaction among the global prompt, layout, and image tokens. This design prevents mis-association between different regions and their descriptions while enabling sufficient injection of layout constraints into the generation process. To further enhance generation quality and layout accuracy, we incorporate Group Relative Policy Optimization (GRPO) based post-training scheme with specially designed layout reward functions for next-set-based AR models. Experimental results demonstrate that SMARLI is able to seamlessly integrate layout tokens with text and image tokens without compromising generation quality. It achieves superior layoutaware control while maintaining the structural simplicity and generation efficiency of AR models.