Understanding Knowledge Gaps in Visual Question Answering: Implications for Gap Identification and Testing
This work addresses the need for more balanced VQA datasets to reduce system dependency on biases, though it is incremental as it builds on existing methods for gap identification.
The authors tackled the problem of quantifying and addressing knowledge gaps in Visual Question Answering (VQA) by using a taxonomy to tag questions and introducing a targeted question generation model to reduce skew in question distributions, resulting in increased diversity and reduced bias in datasets.
Visual Question Answering (VQA) systems are tasked with answering natural language questions corresponding to a presented image. Traditional VQA datasets typically contain questions related to the spatial information of objects, object attributes, or general scene questions. Recently, researchers have recognized the need to improve the balance of such datasets to reduce the system's dependency on memorized linguistic features and statistical biases, while aiming for enhanced visual understanding. However, it is unclear whether any latent patterns exist to quantify and explain these failures. As an initial step towards better quantifying our understanding of the performance of VQA models, we use a taxonomy of Knowledge Gaps (KGs) to tag questions with one or more types of KGs. Each Knowledge Gap (KG) describes the reasoning abilities needed to arrive at a resolution. After identifying KGs for each question, we examine the skew in the distribution of questions for each KG. We then introduce a targeted question generation model to reduce this skew, which allows us to generate new types of questions for an image. These new questions can be added to existing VQA datasets to increase the diversity of questions and reduce the skew.