LLM-Free Image Captioning Evaluation in Reference-Flexible Settings
This work addresses the need for neutral and high-performance evaluation metrics in image captioning, which is important for researchers and practitioners in computer vision and natural language processing, though it is incremental as it builds on existing LLM-free approaches.
The paper tackles the problem of automatic evaluation of image captions in both reference-based and reference-free settings, where existing LLM-based metrics lack neutrality and LLM-free metrics have performance issues, and proposes Pearl, an LLM-free supervised metric that outperforms other LLM-free metrics on multiple datasets, achieving top results on Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and FOIL datasets.
We focus on the automatic evaluation of image captions in both reference-based and reference-free settings. Existing metrics based on large language models (LLMs) favor their own generations; therefore, the neutrality is in question. Most LLM-free metrics do not suffer from such an issue, whereas they do not always demonstrate high performance. To address these issues, we propose Pearl, an LLM-free supervised metric for image captioning, which is applicable to both reference-based and reference-free settings. We introduce a novel mechanism that learns the representations of image--caption and caption--caption similarities. Furthermore, we construct a human-annotated dataset for image captioning metrics, that comprises approximately 333k human judgments collected from 2,360 annotators across over 75k images. Pearl outperformed other existing LLM-free metrics on the Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and FOIL datasets in both reference-based and reference-free settings. Our project page is available at https://pearl.kinsta.page/.