VQA-Levels: A Hierarchical Approach for Classifying Questions in VQA
This addresses the problem for VQA researchers by providing a structured dataset to methodically test and advance systems, though it is incremental as it builds on existing benchmark efforts.
The paper tackles the challenge of systematically evaluating Visual Question Answering (VQA) systems by proposing a new hierarchical benchmark dataset called VQA-Levels, which classifies questions into seven levels based on complexity, from low-level features to high-level abstraction, and initial testing shows existing systems perform best on simpler levels (e.g., Level 1 with high success) and worst on more complex ones (e.g., Level 3 with least success).
Designing datasets for Visual Question Answering (VQA) is a difficult and complex task that requires NLP for parsing and computer vision for analysing the relevant aspects of the image for answering the question asked. Several benchmark datasets have been developed by researchers but there are many issues with using them for methodical performance tests. This paper proposes a new benchmark dataset -- a pilot version called VQA-Levels is ready now -- for testing VQA systems systematically and assisting researchers in advancing the field. The questions are classified into seven levels ranging from direct answers based on low-level image features (without needing even a classifier) to those requiring high-level abstraction of the entire image content. The questions in the dataset exhibit one or many of ten properties. Each is categorised into a specific level from 1 to 7. Levels 1 - 3 are directly on the visual content while the remaining levels require extra knowledge about the objects in the image. Each question generally has a unique one or two-word answer. The questions are 'natural' in the sense that a human is likely to ask such a question when seeing the images. An example question at Level 1 is, ``What is the shape of the red colored region in the image?" while at Level 7, it is, ``Why is the man cutting the paper?". Initial testing of the proposed dataset on some of the existing VQA systems reveals that their success is high on Level 1 (low level features) and Level 2 (object classification) questions, least on Level 3 (scene text) followed by Level 6 (extrapolation) and Level 7 (whole scene analysis) questions. The work in this paper will go a long way to systematically analyze VQA systems.