On the Flip Side: Identifying Counterexamples in Visual Question Answering
This work addresses the evaluation of VQA models for researchers, revealing potential limitations in current benchmarks, but it is incremental as it builds on existing VQA frameworks.
The paper tackles the problem of whether visual question answering (VQA) models learn semantic distinctions by introducing a counterexample prediction task, VQA-CX, and finds that while models surpass benchmarks on this task, state-of-the-art VQA models' multimodal representations do not meaningfully contribute, questioning their general reasoning abilities.
Visual question answering (VQA) models respond to open-ended natural language questions about images. While VQA is an increasingly popular area of research, it is unclear to what extent current VQA architectures learn key semantic distinctions between visually-similar images. To investigate this question, we explore a reformulation of the VQA task that challenges models to identify counterexamples: images that result in a different answer to the original question. We introduce two methods for evaluating existing VQA models against a supervised counterexample prediction task, VQA-CX. While our models surpass existing benchmarks on VQA-CX, we find that the multimodal representations learned by an existing state-of-the-art VQA model do not meaningfully contribute to performance on this task. These results call into question the assumption that successful performance on the VQA benchmark is indicative of general visual-semantic reasoning abilities.