Informative Image Captioning with External Sources of Information
This addresses the limitation of current captioning models that produce generic captions, improving informativeness for applications like accessibility or content indexing.
The paper tackles the problem of generating informative image captions by integrating fine-grained entity labels from external sources, resulting in captions that are fluent and include detailed entity mentions.
An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usually trained to generate captions that only contain common object names, thus falling short on an important "informativeness" dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the image in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the output, resulting in captions that are both fluent and informative.