Answer Questions with Right Image Regions: A Visual Attention Regularization Approach
This work improves visual grounding in VQA models for researchers and practitioners by providing a method that does not require expensive human attention supervision, which is an incremental improvement over existing methods.
This paper addresses the issue of irrelevant visual attention in Visual Question Answering (VQA) models, where highlighted image regions often do not correspond to the question and answer. The proposed method, AttReg, identifies essential but ignored image regions and uses a mask-guided learning scheme to regularize visual attention towards these regions, achieving a new state-of-the-art accuracy of 60.00% on VQA-CP v2, an absolute gain of 7.01%.
Visual attention in Visual Question Answering (VQA) targets at locating the right image regions regarding the answer prediction, offering a powerful technique to promote multi-modal understanding. However, recent studies have pointed out that the highlighted image regions from the visual attention are often irrelevant to the given question and answer, leading to model confusion for correct visual reasoning. To tackle this problem, existing methods mostly resort to aligning the visual attention weights with human attentions. Nevertheless, gathering such human data is laborious and expensive, making it burdensome to adapt well-developed models across datasets. To address this issue, in this paper, we devise a novel visual attention regularization approach, namely AttReg, for better visual grounding in VQA. Specifically, AttReg firstly identifies the image regions which are essential for question answering yet unexpectedly ignored (i.e., assigned with low attention weights) by the backbone model. And then a mask-guided learning scheme is leveraged to regularize the visual attention to focus more on these ignored key regions. The proposed method is very flexible and model-agnostic, which can be integrated into most visual attention-based VQA models and require no human attention supervision. Extensive experiments over three benchmark datasets, i.e., VQA-CP v2, VQA-CP v1, and VQA v2, have been conducted to evaluate the effectiveness of AttReg. As a by-product, when incorporating AttReg into the strong baseline LMH, our approach can achieve a new state-of-the-art accuracy of 60.00% with an absolute performance gain of 7.01% on the VQA-CP v2 benchmark dataset...