Benchmark Visual Question Answer Models by using Focus Map
This work addresses the need for better evaluation of attention mechanisms in visual reasoning models, but it is incremental as it builds on existing focus map visualization techniques.
The paper tackles the problem of evaluating the focus maps in visual question answering models by proposing a method to generate questions and compare inferred focus maps with segmentation graphs, showing that the CLEVR-iep model focuses more accurately than end-to-end models on the CLEVR dataset.
Inferring and Executing Programs for Visual Reasoning proposes a model for visual reasoning that consists of a program generator and an execution engine to avoid end-to-end models. To show that the model actually learns which objects to focus on to answer the questions, the authors give a visualization of the norm of the gradient of the sum of the predicted answer scores with respect to the final feature map. However, the authors do not evaluate the efficiency of focus map. This paper purposed a method for evaluating it. We generate several kinds of questions to test different keywords. We infer focus maps from the model by asking these questions and evaluate them by comparing with the segmentation graph. Furthermore, this method can be applied to any model if focus maps can be inferred from it. By evaluating focus map of different models on the CLEVR dataset, we will show that CLEVR-iep model has learned where to focus more than end-to-end models.