QACE: Asking Questions to Evaluate an Image Caption
This addresses the need for better evaluation metrics in computer vision and natural language processing, particularly for image captioning tasks, though it is incremental as it builds on existing question-answering and VQA methods.
The paper tackles the problem of evaluating image captions by proposing QACE, a metric that generates questions from the caption and checks answers against either a reference caption or the source image, with QACE-Img achieving competitive results compared to other reference-less metrics.
In this paper, we propose QACE, a new metric based on Question Answering for Caption Evaluation. QACE generates questions on the evaluated caption and checks its content by asking the questions on either the reference caption or the source image. We first develop QACE-Ref that compares the answers of the evaluated caption to its reference, and report competitive results with the state-of-the-art metrics. To go further, we propose QACE-Img, which asks the questions directly on the image, instead of reference. A Visual-QA system is necessary for QACE-Img. Unfortunately, the standard VQA models are framed as a classification among only a few thousand categories. Instead, we propose Visual-T5, an abstractive VQA system. The resulting metric, QACE-Img is multi-modal, reference-less, and explainable. Our experiments show that QACE-Img compares favorably w.r.t. other reference-less metrics. We will release the pre-trained models to compute QACE.