Phrase-based Image Captioning
This addresses the problem of generating textual descriptions from images for applications in computer vision and NLP, but it is incremental as it simplifies existing methods.
The paper tackles image captioning by generating descriptive sentences from images using a bilinear model to infer phrases and a language model for syntax, achieving results comparable to state-of-the-art on Flickr30k and Microsoft COCO datasets.
Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representation (generated from a previously trained Convolutional Neural Network) and phrases that are used to described them. The system is then able to infer phrases from a given image sample. Based on caption syntax statistics, we propose a simple language model that can produce relevant descriptions for a given test image using the phrases inferred. Our approach, which is considerably simpler than state-of-the-art models, achieves comparable results in two popular datasets for the task: Flickr30k and the recently proposed Microsoft COCO.