WEmbSim: A Simple yet Effective Metric for Image Captioning
This work addresses the problem of accurately evaluating image captions for researchers and developers, providing a new baseline for future metric development.
This paper proposes WEmbSim, a simple metric for image caption evaluation based on the cosine similarity of Mean of Word Embeddings (MOWE). WEmbSim outperforms complex metrics like SPICE, CIDEr, and WMD in system-level correlation with human judgments and achieves the best accuracy in matching human consensus scores for caption pairs among unsupervised methods.
The area of automatic image caption evaluation is still undergoing intensive research to address the needs of generating captions which can meet adequacy and fluency requirements. Based on our past attempts at developing highly sophisticated learning-based metrics, we have discovered that a simple cosine similarity measure using the Mean of Word Embeddings(MOWE) of captions can actually achieve a surprisingly high performance on unsupervised caption evaluation. This inspires our proposed work on an effective metric WEmbSim, which beats complex measures such as SPICE, CIDEr and WMD at system-level correlation with human judgments. Moreover, it also achieves the best accuracy at matching human consensus scores for caption pairs, against commonly used unsupervised methods. Therefore, we believe that WEmbSim sets a new baseline for any complex metric to be justified.