Exploring Semantic Relationships for Unpaired Image Captioning
This addresses the problem of reducing reliance on time-consuming paired datasets for image captioning, offering an incremental improvement over existing methods.
The paper tackles unpaired image captioning by using high-level semantic information to bridge vision and language domains, achieving desirable captions without paired datasets and boosting five strong baselines with up to an 8% improvement in CIDEr score.
Recently, image captioning has aroused great interest in both academic and industrial worlds. Most existing systems are built upon large-scale datasets consisting of image-sentence pairs, which, however, are time-consuming to construct. In addition, even for the most advanced image captioning systems, it is still difficult to realize deep image understanding. In this work, we achieve unpaired image captioning by bridging the vision and the language domains with high-level semantic information. The motivation stems from the fact that the semantic concepts with the same modality can be extracted from both images and descriptions. To further improve the quality of captions generated by the model, we propose the Semantic Relationship Explorer, which explores the relationships between semantic concepts for better understanding of the image. Extensive experiments on MSCOCO dataset show that we can generate desirable captions without paired datasets. Furthermore, the proposed approach boosts five strong baselines under the paired setting, where the most significant improvement in CIDEr score reaches 8%, demonstrating that it is effective and generalizes well to a wide range of models.