Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
This addresses the problem of improving VQA performance by better capturing visual relationships, though it builds incrementally on existing visual relationship models.
The paper tackles the challenge of modeling relational semantics in Visual Question Answering (VQA) by proposing a scene graph reasoning approach, achieving state-of-the-art accuracy of 54.56% on the GQA dataset.
One of the key issues of Visual Question Answering (VQA) is to reason with semantic clues in the visual content under the guidance of the question, how to model relational semantics still remains as a great challenge. To fully capture visual semantics, we propose to reason over a structured visual representation - scene graph, with embedded objects and inter-object relationships. This shows great benefit over vanilla vector representations and implicit visual relationship learning. Based on existing visual relationship models, we propose a visual relationship encoder that projects visual relationships into a learned deep semantic space constrained by visual context and language priors. Upon the constructed graph, we propose a Scene Graph Convolutional Network (SceneGCN) to jointly reason the object properties and relational semantics for the correct answer. We demonstrate the model's effectiveness and interpretability on the challenging GQA dataset and the classical VQA 2.0 dataset, remarkably achieving state-of-the-art 54.56% accuracy on GQA compared to the existing best model.