Evaluating Image Caption via Cycle-consistent Text-to-Image Generation
This addresses the need for more reliable and cost-effective evaluation in image captioning, though it is incremental as it builds on existing reference-free approaches.
The paper tackles the problem of evaluating image captions without relying on costly and subjective reference captions by proposing CAMScore, a cyclic reference-free metric that uses text-to-image generation to circumvent modality gaps. Results show CAMScore achieves superior correlation with human judgments compared to existing metrics across multiple benchmark datasets.
Evaluating image captions typically relies on reference captions, which are costly to obtain and exhibit significant diversity and subjectivity. While reference-free evaluation metrics have been proposed, most focus on cross-modal evaluation between captions and images. Recent research has revealed that the modality gap generally exists in the representation of contrastive learning-based multi-modal systems, undermining the reliability of cross-modality metrics like CLIPScore. In this paper, we propose CAMScore, a cyclic reference-free automatic evaluation metric for image captioning models. To circumvent the aforementioned modality gap, CAMScore utilizes a text-to-image model to generate images from captions and subsequently evaluates these generated images against the original images. Furthermore, to provide fine-grained information for a more comprehensive evaluation, we design a three-level evaluation framework for CAMScore that encompasses pixel-level, semantic-level, and objective-level perspectives. Extensive experiment results across multiple benchmark datasets show that CAMScore achieves a superior correlation with human judgments compared to existing reference-based and reference-free metrics, demonstrating the effectiveness of the framework.