CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
This addresses the issue of semantic fidelity in text-to-image generation for users needing accurate visual representations from complex prompts, representing an incremental improvement over existing tailored solutions.
The paper tackles the problem of text-to-image diffusion models failing to faithfully represent semantic intent in prompts, such as overlooking objects, by introducing a contrastive approach that segregates objects in attention maps while keeping related attributes close. The method demonstrates versatility and efficiency across various scenarios and models like Stable Diffusion and Imagen, with publicly shared source code.
Images produced by text-to-image diffusion models might not always faithfully represent the semantic intent of the provided text prompt, where the model might overlook or entirely fail to produce certain objects. Existing solutions often require customly tailored functions for each of these problems, leading to sub-optimal results, especially for complex prompts. Our work introduces a novel perspective by tackling this challenge in a contrastive context. Our approach intuitively promotes the segregation of objects in attention maps while also maintaining that pairs of related attributes are kept close to each other. We conduct extensive experiments across a wide variety of scenarios, each involving unique combinations of objects, attributes, and scenes. These experiments effectively showcase the versatility, efficiency, and flexibility of our method in working with both latent and pixel-based diffusion models, including Stable Diffusion and Imagen. Moreover, we publicly share our source code to facilitate further research.