Visual Question Answering based on Local-Scene-Aware Referring Expression Generation
This addresses the limitation of existing VQA methods that rely on simple object relationships, offering a more robust solution for complex scene understanding, though it is incremental as it builds on prior work with a novel integration.
The paper tackled the problem of insufficient scene representation in visual question answering by generating local-scene-aware referring expressions to enrich image descriptions, resulting in outperforming state-of-the-art methods on the VQA v2 dataset with improved quantitative and qualitative results.
Visual question answering requires a deep understanding of both images and natural language. However, most methods mainly focus on visual concept; such as the relationships between various objects. The limited use of object categories combined with their relationships or simple question embedding is insufficient for representing complex scenes and explaining decisions. To address this limitation, we propose the use of text expressions generated for images, because such expressions have few structural constraints and can provide richer descriptions of images. The generated expressions can be incorporated with visual features and question embedding to obtain the question-relevant answer. A joint-embedding multi-head attention network is also proposed to model three different information modalities with co-attention. We quantitatively and qualitatively evaluated the proposed method on the VQA v2 dataset and compared it with state-of-the-art methods in terms of answer prediction. The quality of the generated expressions was also evaluated on the RefCOCO, RefCOCO+, and RefCOCOg datasets. Experimental results demonstrate the effectiveness of the proposed method and reveal that it outperformed all of the competing methods in terms of both quantitative and qualitative results.