ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
For researchers evaluating MLLM captioning, this provides a more diverse benchmark and an automated metric that reduces reliance on human annotation.
The paper introduces ICBench, a large-scale image captioning benchmark with 40K captions from 10 MLLMs across 12 categories, and proposes ITIScore, an automated metric based on image-to-text-to-image reconstruction consistency. ITIScore shows strong alignment with human judgments and robust zero-shot generalization.
Recent advances in multimodal large language models (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing image captioning benchmarks typically suffer from limited diversity in caption length, the absence of recent advanced MLLMs, and insufficient human annotations, which potentially introduces bias and limits the ability to comprehensively assess the performance of modern MLLMs. To address these limitations, we present a new large-scale image captioning benchmark, termed, ICBench, which covers 12 content categories and consists of both short and long captions generated by 10 advanced MLLMs on 2K images, resulting in 40K captions in total. We conduct extensive human subjective studies to obtain mean opinion scores (MOSs) across fine-grained evaluation dimensions, where short captions are assessed in terms of fluency, relevance, and conciseness, while long captions are evaluated based on fluency, relevance, and completeness. Furthermore, we propose an automated evaluation metric, \textbf{ITIScore}, based on an image-to-text-to-image framework, which measures caption quality through reconstruction consistency. Experimental results demonstrate strong alignment between our automatic metric and human judgments, as well as robust zero-shot generalization ability on other public captioning datasets. Both the dataset and model will be released upon publication.