Word to Sentence Visual Semantic Similarity for Caption Generation: Lessons Learned
This work addresses the challenge of generating more accurate and relevant captions for image-caption systems, which is incremental as it builds on existing methods with a novel post-processing approach.
The paper tackles the problem of improving image caption generation by selecting the most visually relevant caption from model outputs rather than the most probable one, using a visual semantic similarity measure at word and sentence levels, and reports that it can be applied as a post-processing method to any caption system.
This paper focuses on enhancing the captions generated by image-caption generation systems. We propose an approach for improving caption generation systems by choosing the most closely related output to the image rather than the most likely output produced by the model. Our model revises the language generation output beam search from a visual context perspective. We employ a visual semantic measure in a word and sentence level manner to match the proper caption to the related information in the image. The proposed approach can be applied to any caption system as a post-processing based method.