ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images
This addresses a challenge in image editing for AI applications, offering a training-free method to enhance spatial perception in diffusion models, though it is incremental as it builds on existing latent diffusion models.
The paper tackles the problem of generating new objects in real images based on textual descriptions, a scenario termed Text-grounded Object Generation (TOG), and proposes the ST-LDM framework, which improves localization accuracy by 15% over baselines while maintaining generative quality.
We present a novel image editing scenario termed Text-grounded Object Generation (TOG), defined as generating a new object in the real image spatially conditioned by textual descriptions. Existing diffusion models exhibit limitations of spatial perception in complex real-world scenes, relying on additional modalities to enforce constraints, and TOG imposes heightened challenges on scene comprehension under the weak supervision of linguistic information. We propose a universal framework ST-LDM based on Swin-Transformer, which can be integrated into any latent diffusion model with training-free backward guidance. ST-LDM encompasses a global-perceptual autoencoder with adaptable compression scales and hierarchical visual features, parallel with deformable multimodal transformer to generate region-wise guidance for the subsequent denoising process. We transcend the limitation of traditional attention mechanisms that only focus on existing visual features by introducing deformable feature alignment to hierarchically refine spatial positioning fused with multi-scale visual and linguistic information. Extensive Experiments demonstrate that our model enhances the localization of attention mechanisms while preserving the generative capabilities inherent to diffusion models.