An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
This addresses the problem of improving accuracy and explainability in Visual QA for AI researchers, though it is incremental as it builds on existing graph network methods.
The paper tackles Visual Question Answering by using scene graphs derived from images and graph networks for structured reasoning, demonstrating that this approach can outperform state-of-the-art algorithms with a cleaner architecture and offer interpretability.
Visual question answering (Visual QA) has attracted significant attention these years. While a variety of algorithms have been proposed, most of them are built upon different combinations of image and language features as well as multi-modal attention and fusion. In this paper, we investigate an alternative approach inspired by conventional QA systems that operate on knowledge graphs. Specifically, we investigate the use of scene graphs derived from images for Visual QA: an image is abstractly represented by a graph with nodes corresponding to object entities and edges to object relationships. We adapt the recently proposed graph network (GN) to encode the scene graph and perform structured reasoning according to the input question. Our empirical studies demonstrate that scene graphs can already capture essential information of images and graph networks have the potential to outperform state-of-the-art Visual QA algorithms but with a much cleaner architecture. By analyzing the features generated by GNs we can further interpret the reasoning process, suggesting a promising direction towards explainable Visual QA.