A Practical Investigation of Spatially-Controlled Image Generation with Transformers
This work provides practical insights for practitioners developing transformer-based spatially-controlled image generation systems, clarifying literature and addressing knowledge gaps, but it is incremental as it focuses on comparisons and enhancements rather than introducing a new paradigm.
The paper tackled the problem of enabling spatially-controlled image generation with transformers by performing controlled experiments across diffusion-based, flow-based, and autoregressive models on ImageNet, establishing control token prefilling as a baseline and showing that classifier-free guidance and softmax truncation improve control-generation consistency.
Enabling image generation models to be spatially controlled is an important area of research, empowering users to better generate images according to their own fine-grained specifications via e.g. edge maps, poses. Although this task has seen impressive improvements in recent times, a focus on rapidly producing stronger models has come at the cost of detailed and fair scientific comparison. Differing training data, model architectures and generation paradigms make it difficult to disentangle the factors contributing to performance. Meanwhile, the motivations and nuances of certain approaches become lost in the literature. In this work, we aim to provide clear takeaways across generation paradigms for practitioners wishing to develop transformer-based systems for spatially-controlled generation, clarifying the literature and addressing knowledge gaps. We perform controlled experiments on ImageNet across diffusion-based/flow-based and autoregressive (AR) models. First, we establish control token prefilling as a simple, general and performant baseline approach for transformers. We then investigate previously underexplored sampling time enhancements, showing that extending classifier-free guidance to control, as well as softmax truncation, have a strong impact on control-generation consistency. Finally, we re-clarify the motivation of adapter-based approaches, demonstrating that they mitigate "forgetting" and maintain generation quality when trained on limited downstream data, but underperform full training in terms of generation-control consistency.