DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning
This work addresses the challenge of reliable automatic evaluation for image captioning, particularly in handling hallucinations, which is crucial for researchers and developers in computer vision and NLP.
The authors tackled the problem of evaluating image captions with a focus on robustness against hallucinations, proposing DENEB, a supervised metric that achieved state-of-the-art performance on multiple datasets, including FOIL and Flickr8K-Expert.
In this work, we address the challenge of developing automatic evaluation metrics for image captioning, with a particular focus on robustness against hallucinations. Existing metrics are often inadequate for handling hallucinations, primarily due to their limited ability to compare candidate captions with multifaceted reference captions. To address this shortcoming, we propose DENEB, a novel supervised automatic evaluation metric specifically robust against hallucinations. DENEB incorporates the Sim-Vec Transformer, a mechanism that processes multiple references simultaneously, thereby efficiently capturing the similarity between an image, a candidate caption, and reference captions. To train DENEB, we construct the diverse and balanced Nebula dataset comprising 32,978 images, paired with human judgments provided by 805 annotators. We demonstrated that DENEB achieves state-of-the-art performance among existing LLM-free metrics on the FOIL, Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and PASCAL-50S datasets, validating its effectiveness and robustness against hallucinations.