TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models
This provides a standardized evaluation framework for researchers and industry in text-to-image generation, though it is incremental as it builds on existing metrics and datasets.
The authors tackled the challenge of evaluating text-to-image models by creating TeTIm-Eval, a curated dataset with ten categories, and applied CLIP-score and human evaluation to compare models like DALLE2 and Stable Diffusion, finding that human judgment accuracy aligned with CLIP-score.
Evaluating and comparing text-to-image models is a challenging problem. Significant advances in the field have recently been made, piquing interest of various industrial sectors. As a consequence, a gold standard in the field should cover a variety of tasks and application contexts. In this paper a novel evaluation approach is experimented, on the basis of: (i) a curated data set, made by high-quality royalty-free image-text pairs, divided into ten categories; (ii) a quantitative metric, the CLIP-score, (iii) a human evaluation task to distinguish, for a given text, the real and the generated images. The proposed method has been applied to the most recent models, i.e., DALLE2, Latent Diffusion, Stable Diffusion, GLIDE and Craiyon. Early experimental results show that the accuracy of the human judgement is fully coherent with the CLIP-score. The dataset has been made available to the public.