VCRScore: Image captioning metric based on V\&L Transformers, CLIP, and precision-recall
This addresses the need for better evaluation methods in vision-language research, though it is incremental as it builds on existing metric frameworks.
The authors tackled the problem of outdated evaluation metrics in image captioning by proposing VCRScore, a new metric based on V&L Transformers and CLIP, which outperformed existing metrics like BLEU and CIDEr on a human-labeled dataset.
Image captioning has become an essential Vision & Language research task. It is about predicting the most accurate caption given a specific image or video. The research community has achieved impressive results by continuously proposing new models and approaches to improve the overall model's performance. Nevertheless, despite increasing proposals, the performance metrics used to measure their advances have remained practically untouched through the years. A probe of that, nowadays metrics like BLEU, METEOR, CIDEr, and ROUGE are still very used, aside from more sophisticated metrics such as BertScore and ClipScore. Hence, it is essential to adjust how are measure the advances, limitations, and scopes of the new image captioning proposals, as well as to adapt new metrics to these new advanced image captioning approaches. This work proposes a new evaluation metric for the image captioning problem. To do that, first, it was generated a human-labeled dataset to assess to which degree the captions correlate with the image's content. Taking these human scores as ground truth, we propose a new metric, and compare it with several well-known metrics, from classical to newer ones. Outperformed results were also found, and interesting insights were presented and discussed.