Scene-based Factored Attention for Image Captioning
This work addresses caption generation issues for multimedia applications, but it is incremental as it builds on existing encoder-decoder frameworks with attention mechanisms.
The paper tackles the problem of sentence bias in image captioning by introducing a scene-based factored attention module that embeds scene concepts to attend visual information, resulting in improved performance on the Microsoft COCO benchmark across various metrics.
Image captioning has attracted ever-increasing research attention in the multimedia community. To this end, most cutting-edge works rely on an encoder-decoder framework with attention mechanisms, which have achieved remarkable progress. However, such a framework does not consider scene concepts to attend visual information, which leads to sentence bias in caption generation and defects the performance correspondingly. We argue that such scene concepts capture higher-level visual semantics and serve as an important cue in describing images. In this paper, we propose a novel scene-based factored attention module for image captioning. Specifically, the proposed module first embeds the scene concepts into factored weights explicitly and attends the visual information extracted from the input image. Then, an adaptive LSTM is used to generate captions for specific scene types. Experimental results on Microsoft COCO benchmark show that the proposed scene-based attention module improves model performance a lot, which outperforms the state-of-the-art approaches under various evaluation metrics.