Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models
This addresses the problem of poor text-image alignment for users of text-to-image models, though it is an incremental improvement focusing on attention map alignment.
The paper tackled semantic misalignment in text-to-image diffusion models, such as incorrect attribute binding and object neglect, by proposing an object-conditioned Energy-Based Attention Map Alignment (EBAMA) method, which demonstrated superior performance in qualitative and quantitative experiments over previous methods.
Text-to-image diffusion models have shown great success in generating high-quality text-guided images. Yet, these models may still fail to semantically align generated images with the provided text prompts, leading to problems like incorrect attribute binding and/or catastrophic object neglect. Given the pervasive object-oriented structure underlying text prompts, we introduce a novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method to address the aforementioned problems. We show that an object-centric attribute binding loss naturally emerges by approximately maximizing the log-likelihood of a $z$-parameterized energy-based model with the help of the negative sampling technique. We further propose an object-centric intensity regularizer to prevent excessive shifts of objects attention towards their attributes. Extensive qualitative and quantitative experiments, including human evaluation, on several challenging benchmarks demonstrate the superior performance of our method over previous strong counterparts. With better aligned attention maps, our approach shows great promise in further enhancing the text-controlled image editing ability of diffusion models.