Knowing Where to Look? Analysis on Attention of Visual Question Answering System
This work helps researchers identify challenges to improve VQA systems, but it is incremental as it analyzes existing methods without proposing new solutions.
The paper analyzed attention maps from two state-of-the-art Visual Question Answering systems to identify their limitations, finding that they are sensitive to features and perform poorly on counting and multi-object questions.
Attention mechanisms have been widely used in Visual Question Answering (VQA) solutions due to their capacity to model deep cross-domain interactions. Analyzing attention maps offers us a perspective to find out limitations of current VQA systems and an opportunity to further improve them. In this paper, we select two state-of-the-art VQA approaches with attention mechanisms to study their robustness and disadvantages by visualizing and analyzing their estimated attention maps. We find that both methods are sensitive to features, and simultaneously, they perform badly for counting and multi-object related questions. We believe that the findings and analytical method will help researchers identify crucial challenges on the way to improve their own VQA systems.