OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts
This work addresses caption generation for abstract scenes, which is incremental as it builds on existing sequence-to-sequence methods for a specific domain.
The paper tackled the problem of generating captions for abstract scenes using only object layouts, proposing OBJ2TEXT, a sequence-to-sequence model that encodes objects and locations to produce coherent descriptions. It improved an image captioning model's CIDEr score from 0.863 to 0.950 on the MS-COCO benchmark.
Generating captions for images is a task that has recently received considerable attention. In this work we focus on caption generation for abstract scenes, or object layouts where the only information provided is a set of objects and their locations. We propose OBJ2TEXT, a sequence-to-sequence model that encodes a set of objects and their locations as an input sequence using an LSTM network, and decodes this representation using an LSTM language model. We show that our model, despite encoding object layouts as a sequence, can represent spatial relationships between objects, and generate descriptions that are globally coherent and semantically relevant. We test our approach in a task of object-layout captioning by using only object annotations as inputs. We additionally show that our model, combined with a state-of-the-art object detector, improves an image captioning model from 0.863 to 0.950 (CIDEr score) in the test benchmark of the standard MS-COCO Captioning task.