Attention Guided Semantic Relationship Parsing for Visual Question Answering
This addresses the challenge of multi-modal understanding in VQA by improving relationship parsing, though it appears incremental as it builds on existing attention mechanisms.
The paper tackles the problem of representing inter-object relationships in Visual Question Answering (VQA) by proposing a semantic relationship parser that generates semantic feature vectors for subject-predicate-object triplets, achieving a ~25% accuracy gain over state-of-the-art models in an oracle setting on the GQA dataset.
Humans explain inter-object relationships with semantic labels that demonstrate a high-level understanding required to perform complex Vision-Language tasks such as Visual Question Answering (VQA). However, existing VQA models represent relationships as a combination of object-level visual features which constrain a model to express interactions between objects in a single domain, while the model is trying to solve a multi-modal task. In this paper, we propose a general purpose semantic relationship parser which generates a semantic feature vector for each subject-predicate-object triplet in an image, and a Mutual and Self Attention (MSA) mechanism that learns to identify relationship triplets that are important to answer the given question. To motivate the significance of semantic relationships, we show an oracle setting with ground-truth relationship triplets, where our model achieves a ~25% accuracy gain over the closest state-of-the-art model on the challenging GQA dataset. Further, with our semantic parser, we show that our model outperforms other comparable approaches on VQA and GQA datasets.