PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning
This addresses a critical weakness in evaluation metrics for multilingual image captioning, though it is incremental as it builds on CLIPScore.
The paper tackles the vulnerability of automatic evaluation metrics to lexical perturbations in multilingual image captioning by proposing PR-MCS, a reference-free metric that significantly outperforms baselines in robustness across five languages.
Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning. This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages. To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text. To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3, 000 images in five languages. In our experiments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages, proving that PR-MCS is highly robust to lexical perturbations.