Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation
This addresses a critical gap in evaluating text-to-image models for real-world knowledge alignment, which is important for users relying on accurate and contextually appropriate image generation, though it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem that text-to-image generation models often fail to align generated images with real-world knowledge beyond explicit prompts, and introduces a benchmark (ABP) and metric (ABPScore) to evaluate this, finding state-of-the-art models like GPT-4o have limitations, with a training-free strategy improving scores by about 43% on challenging samples.
Recent text-to-image (T2I) generation models have advanced significantly, enabling the creation of high-fidelity images from textual prompts. However, existing evaluation benchmarks primarily focus on the explicit alignment between generated images and prompts, neglecting the alignment with real-world knowledge beyond prompts. To address this gap, we introduce Align Beyond Prompts (ABP), a comprehensive benchmark designed to measure the alignment of generated images with real-world knowledge that extends beyond the explicit user prompts. ABP comprises over 2,000 meticulously crafted prompts, covering real-world knowledge across six distinct scenarios. We further introduce ABPScore, a metric that utilizes existing Multimodal Large Language Models (MLLMs) to assess the alignment between generated images and world knowledge beyond prompts, which demonstrates strong correlations with human judgments. Through a comprehensive evaluation of 8 popular T2I models using ABP, we find that even state-of-the-art models, such as GPT-4o, face limitations in integrating simple real-world knowledge into generated images. To mitigate this issue, we introduce a training-free strategy within ABP, named Inference-Time Knowledge Injection (ITKI). By applying this strategy to optimize 200 challenging samples, we achieved an improvement of approximately 43% in ABPScore. The dataset and code are available in https://github.com/smile365317/ABP.