CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
This work addresses the need for explainable AI in VQA by creating a dataset for training and evaluating explanation models, but it is incremental as it builds upon an existing dataset.
The authors tackled the problem of generating natural language explanations for Visual Question Answering (VQA) by introducing the CLEVR-X dataset, which extends CLEVR with structured textual explanations derived from scene graphs, and they provided baseline results and analysis using state-of-the-art frameworks.
Providing explanations in the context of Visual Question Answering (VQA) presents a fundamental problem in machine learning. To obtain detailed insights into the process of generating natural language explanations for VQA, we introduce the large-scale CLEVR-X dataset that extends the CLEVR dataset with natural language explanations. For each image-question pair in the CLEVR dataset, CLEVR-X contains multiple structured textual explanations which are derived from the original scene graphs. By construction, the CLEVR-X explanations are correct and describe the reasoning and visual information that is necessary to answer a given question. We conducted a user study to confirm that the ground-truth explanations in our proposed dataset are indeed complete and relevant. We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset. Furthermore, we provide a detailed analysis of the explanation generation quality for different question and answer types. Additionally, we study the influence of using different numbers of ground-truth explanations on the convergence of natural language generation (NLG) metrics. The CLEVR-X dataset is publicly available at \url{https://explainableml.github.io/CLEVR-X/}.