Learning to Agree on Vision Attention for Visual Commonsense Reasoning
This addresses a specific bottleneck in VCR models for researchers in visual reasoning, though it is incremental as it builds on existing attention and Transformer models.
The paper tackles the problem of Visual Commonsense Reasoning (VCR) by proposing a novel visual attention alignment method to unify the sequential answering and rationale prediction processes, achieving a considerable improvement over baseline methods on the VCR benchmark dataset.
Visual Commonsense Reasoning (VCR) remains a significant yet challenging research problem in the realm of visual reasoning. A VCR model generally aims at answering a textual question regarding an image, followed by the rationale prediction for the preceding answering process. Though these two processes are sequential and intertwined, existing methods always consider them as two independent matching-based instances. They, therefore, ignore the pivotal relationship between the two processes, leading to sub-optimal model performance. This paper presents a novel visual attention alignment method to efficaciously handle these two processes in a unified framework. To achieve this, we first design a re-attention module for aggregating the vision attention map produced in each process. Thereafter, the resultant two sets of attention maps are carefully aligned to guide the two processes to make decisions based on the same image regions. We apply this method to both conventional attention and the recent Transformer models and carry out extensive experiments on the VCR benchmark dataset. The results demonstrate that with the attention alignment module, our method achieves a considerable improvement over the baseline methods, evidently revealing the feasibility of the coupling of the two processes as well as the effectiveness of the proposed method.