Scene Graph Generation for Better Image Captioning?
This work addresses the problem of generating more accurate and descriptive image captions for applications in computer vision and natural language processing, representing an incremental improvement by integrating scene graphs into the captioning pipeline.
The paper tackles image captioning by generating scene graphs from images to model objects and visual relationships, then using a graph-to-text model for caption generation, resulting in improved performance over end-to-end models on BLEU and METEOR metrics.
We investigate the incorporation of visual relationships into the task of supervised image caption generation by proposing a model that leverages detected objects and auto-generated visual relationships to describe images in natural language. To do so, we first generate a scene graph from raw image pixels by identifying individual objects and visual relationships between them. This scene graph then serves as input to our graph-to-text model, which generates the final caption. In contrast to previous approaches, our model thus explicitly models the detection of objects and visual relationships in the image. For our experiments we construct a new dataset from the intersection of Visual Genome and MS COCO, consisting of images with both a corresponding gold scene graph and human-authored caption. Our results show that our methods outperform existing state-of-the-art end-to-end models that generate image descriptions directly from raw input pixels when compared in terms of the BLEU and METEOR evaluation metrics.