InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation
This work addresses the need for more informative and explainable evaluation in image captioning research, offering a tool for benchmarking and advancing the field.
The authors tackled the problem of evaluating image captions by proposing InfoMetIC, a metric that provides both fine-grained feedback on incorrect words and unmentioned regions and coarse-grained scores, achieving significantly better correlation with human judgments than existing metrics.
Automatic image captioning evaluation is critical for benchmarking and promoting advances in image captioning research. Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative. Instead, we humans can easily identify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality. To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC). Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level. The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than existing metrics on multiple benchmarks. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation. Our code and datasets are publicly available at https://github.com/HAWLYQ/InfoMetIC.