DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioning
This addresses the problem of reliable automated evaluation for image captioning, particularly in diverse domains, though it is incremental as it builds on existing LVLM capabilities.
The paper tackles the challenge of robust image caption evaluation under domain shifts by introducing DISCODE, a finetuning-free method that uses a test-time adaptive approach with a Gaussian prior loss, achieving state-of-the-art performance across multiple benchmarks including a new six-domain benchmark.
Large vision-language models (LVLMs) have shown impressive performance across a broad range of multimodal tasks. However, robust image caption evaluation using LVLMs remains challenging, particularly under domain-shift scenarios. To address this issue, we introduce the Distribution-Aware Score Decoder (DISCODE), a novel finetuning-free method that generates robust evaluation scores better aligned with human judgments across diverse domains. The core idea behind DISCODE lies in its test-time adaptive evaluation approach, which introduces the Adaptive Test-Time (ATT) loss, leveraging a Gaussian prior distribution to improve robustness in evaluation score estimation. This loss is efficiently minimized at test time using an analytical solution that we derive. Furthermore, we introduce the Multi-domain Caption Evaluation (MCEval) benchmark, a new image captioning evaluation benchmark covering six distinct domains, designed to assess the robustness of evaluation metrics. In our experiments, we demonstrate that DISCODE achieves state-of-the-art performance as a reference-free evaluation metric across MCEval and four representative existing benchmarks.