Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
This work addresses the interpretability gap in VQA for AI researchers, though it is incremental as it primarily provides a dataset and analysis without proposing new methods.
The study investigated whether deep learning models and humans focus on the same image regions in Visual Question Answering (VQA), finding that current attention models do not align with human attention patterns.
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.