Better Understanding Hierarchical Visual Relationship for Image Caption
This work addresses the challenge of improving image captioning accuracy by better modeling visual relationships, which is incremental as it builds on existing encoder-decoder frameworks.
The paper tackled the problem of capturing hierarchical visual relationships for image captioning by proposing a CNN plus Graph Convolutional Network architecture that integrates semantic and spatial relationships into an encoder, resulting in outperforming previous state-of-the-art models on the COCO dataset across all evaluation metrics.
The Convolutional Neural Network (CNN) has been the dominant image feature extractor in computer vision for years. However, it fails to get the relationship between images/objects and their hierarchical interactions which can be helpful for representing and describing an image. In this paper, we propose a new design for image caption under a general encoder-decoder framework. It takes into account the hierarchical interactions between different abstraction levels of visual information in the images and their bounding-boxes. Specifically, we present CNN plus Graph Convolutional Network (GCN) architecture that novelly integrates both semantic and spatial visual relationships into image encoder. The representations of regions in an image and the connections between images are refined by leveraging graph structure through GCN. With the learned multi-level features, our model capitalizes on the Transformer-based decoder for description generation. We conduct experiments on the COCO image captioning dataset. Evaluations show that our proposed model outperforms the previous state-of-the-art models in the task of image caption, leading to a better performance in terms of all evaluation metrics.