Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content
This work addresses the need for targeted guidance in improving generative models for AI-generated images, though it is incremental as it builds on existing evaluation frameworks by adding fine-grained dimensions.
The paper tackles the problem of coarse quality assessment for AI-generated images by introducing Q-Real, a dataset for fine-grained evaluation of realism and plausibility, consisting of 3,088 images with annotations for entities, judgment questions, and attribution descriptions, and demonstrates its high quality and significance through experiments on multi-modal large language models.
Quality assessment of AI-generated content is crucial for evaluating model capability and guiding model optimization. However, most existing quality assessment datasets and models provide only a single quality score, which is too coarse to offer targeted guidance for improving generative models. In current applications of AI-generated images, realism and plausibility are two critical dimensions, and with the emergence of unified generation-understanding models, fine-grained evaluation along these dimensions becomes especially effective for improving generative performance. Therefore, we introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images. Q-Real consists of 3,088 images generated by popular text-to-image models. For each image, we annotate the locations of major entities and provide a set of judgment questions and attribution descriptions for these along the dimensions of realism and plausibility. Considering that recent advances in multi-modal large language models (MLLMs) enable fine-grained evaluation of AI-generated images, we construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning. Finally, to enhance MLLM capabilities, we design a fine-tuning framework and conduct experiments on multiple MLLMs using our dataset. Experimental results demonstrate the high quality and significance of our dataset and the comprehensiveness of the benchmark. Dataset and code will be released upon publication.