T2I-FineEval: Fine-Grained Compositional Metric for Text-to-Image Evaluation
This addresses the need for better evaluation tools in text-to-image generation, particularly for researchers and developers working on compositional accuracy, though it is incremental as it builds on prior VQA-based approaches.
The paper tackles the problem of evaluating text-to-image models on compositional prompts, where existing metrics like CLIPScore and VQA-based methods fail to fully capture attributes and spatial relationships. It proposes a novel metric that decomposes images and texts into fine-grained components, outperforming previous state-of-the-art metrics.
Although recent text-to-image generative models have achieved impressive performance, they still often struggle with capturing the compositional complexities of prompts including attribute binding, and spatial relationships between different entities. This misalignment is not revealed by common evaluation metrics such as CLIPScore. Recent works have proposed evaluation metrics that utilize Visual Question Answering (VQA) by decomposing prompts into questions about the generated image for more robust compositional evaluation. Although these methods align better with human evaluations, they still fail to fully cover the compositionality within the image. To address this, we propose a novel metric that breaks down images into components, and texts into fine-grained questions about the generated image for evaluation. Our method outperforms previous state-of-the-art metrics, demonstrating its effectiveness in evaluating text-to-image generative models. Code is available at https://github.com/hadi-hosseini/ T2I-FineEval.