Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
This work addresses the interpretability gap in VQA for researchers and practitioners, but it is incremental as it primarily provides a dataset and evaluation without proposing new models.
The paper tackled the problem of understanding whether deep networks and humans attend to similar regions in Visual Question Answering (VQA) by introducing the VQA-HAT dataset through novel attention-annotation interfaces, and found that current VQA models do not look at the same regions as humans.
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.