Guiding Attention using Partial-Order Relationships for Image Captioning
This work addresses the challenge of improving caption accuracy for image captioning systems, but it is incremental as it builds on existing attention-based approaches.
The paper tackles the problem of generating more visually accurate image captions by introducing a guided attention network that exploits relationships between visual scenes and text descriptions using spatial features, high-level topics, and temporal context, embedded in an ordered space with a pairwise ranking objective; experimental results on the MSCOCO dataset show competitiveness with state-of-the-art models on various metrics.
The use of attention models for automated image captioning has enabled many systems to produce accurate and meaningful descriptions for images. Over the years, many novel approaches have been proposed to enhance the attention process using different feature representations. In this paper, we extend this approach by creating a guided attention network mechanism, that exploits the relationship between the visual scene and text-descriptions using spatial features from the image, high-level information from the topics, and temporal context from caption generation, which are embedded together in an ordered embedding space. A pairwise ranking objective is used for training this embedding space which allows similar images, topics and captions in the shared semantic space to maintain a partial order in the visual-semantic hierarchy and hence, helps the model to produce more visually accurate captions. The experimental results based on MSCOCO dataset shows the competitiveness of our approach, with many state-of-the-art models on various evaluation metrics.