Progressive Compositionality in Text-to-Image Generative Models
This addresses a key limitation in text-to-image generation for applications requiring precise object-attribute compositions, though it is incremental as it builds on existing contrastive learning and dataset curation methods.
The paper tackles the problem of text-to-image diffusion models struggling with compositional relationships by generating a contrastive dataset of 15k image pairs and proposing a multi-stage curriculum for contrastive learning, resulting in improved performance on compositional benchmarks.
Despite the impressive text-to-image (T2I) synthesis capabilities of diffusion models, they often struggle to understand compositional relationships between objects and attributes, especially in complex settings. Existing solutions have tackled these challenges by optimizing the cross-attention mechanism or learning from the caption pairs with minimal semantic changes. However, can we generate high-quality complex contrastive images that diffusion models can directly discriminate based on visual representations? In this work, we leverage large-language models (LLMs) to compose realistic, complex scenarios and harness Visual-Question Answering (VQA) systems alongside diffusion models to automatically curate a contrastive dataset, ConPair, consisting of 15k pairs of high-quality contrastive images. These pairs feature minimal visual discrepancies and cover a wide range of attribute categories, especially complex and natural scenarios. To learn effectively from these error cases, i.e., hard negative images, we propose EvoGen, a new multi-stage curriculum for contrastive learning of diffusion models. Through extensive experiments across a wide range of compositional scenarios, we showcase the effectiveness of our proposed framework on compositional T2I benchmarks.