Attention Mechanism based Cognition-level Scene Understanding
This addresses the problem of cognition-level scene understanding for applications like visual question answering and automated vehicle systems, but it appears incremental as it builds on attention mechanisms for a specific task.
The paper tackles the Visual Commonsense Reasoning (VCR) task, which requires predicting answers and rationales from question-image inputs using real-world inference, by proposing a parallel attention-based network (PAVCR) that fuses visual-textual information efficiently; it shows significant improvements over existing methods on the benchmark dataset.
Given a question-image input, the Visual Commonsense Reasoning (VCR) model can predict an answer with the corresponding rationale, which requires inference ability from the real world. The VCR task, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge, is a cognition-level scene understanding task. The VCR task has aroused researchers' interest due to its wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and losing information in long sequences. In this paper, we propose a parallel attention-based cognitive VCR network PAVCR, which fuses visual-textual information efficiently and encodes semantic information in parallel to enable the model to capture rich information for cognition-level inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning.