On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
This addresses the issue of overfitting to spurious correlations in VQA for researchers and practitioners, though it is incremental as it focuses on a specific dataset and evaluation metric.
The authors tackled the problem of poor generalization in Visual Question Answering (VQA) methods by introducing a bilingual scene-text VQA dataset and an evaluation process that measures reasoning ability to penalize coincidentally correct answers.
Visual Question Answering (VQA) methods have made incredible progress, but suffer from a failure to generalize. This is visible in the fact that they are vulnerable to learning coincidental correlations in the data rather than deeper relations between image content and ideas expressed in language. We present a dataset that takes a step towards addressing this problem in that it contains questions expressed in two languages, and an evaluation process that co-opts a well understood image-based metric to reflect the method's ability to reason. Measuring reasoning directly encourages generalization by penalizing answers that are coincidentally correct. The dataset reflects the scene-text version of the VQA problem, and the reasoning evaluation can be seen as a text-based version of a referring expression challenge. Experiments and analysis are provided that show the value of the dataset.