End-to-End Learning Using Cycle Consistency for Image-to-Caption Transformations
This work addresses the challenge of producing accurate and informative captions for images, which is important for applications in computer vision and natural language processing, but it appears incremental as it builds on existing cycle-consistency concepts.
The paper tackles the problem of generating faithful image captions by ensuring they contain sufficient information to reconstruct the original image, using an end-to-end learning approach with cycle-consistency loss. The results show the method is effective, as demonstrated through automatic evaluation and crowdsourcing, though no specific numbers are provided.
So far, research to generate captions from images has been carried out from the viewpoint that a caption holds sufficient information for an image. If it is possible to generate an image that is close to the input image from a generated caption, i.e., if it is possible to generate a natural language caption containing sufficient information to reproduce the image, then the caption is considered to be faithful to the image. To make such regeneration possible, learning using the cycle-consistency loss is effective. In this study, we propose a method of generating captions by learning end-to-end mutual transformations between images and texts. To evaluate our method, we perform comparative experiments with and without the cycle consistency. The results are evaluated by an automatic evaluation and crowdsourcing, demonstrating that our proposed method is effective.