BloomVQA: Assessing Hierarchical Multi-modal Comprehension
This addresses the need for better evaluation of multi-modal AI models in education and cognitive tasks, though it is incremental as it builds on existing VQA benchmarks with a new dataset and analysis.
The authors tackled the problem of evaluating vision-language models on comprehension tasks by introducing BloomVQA, a dataset based on Bloom's Taxonomy, and found that models show up to a 38.0% drop in accuracy on advanced comprehension tasks compared to low-level ones, with GPT-4V improving across levels but sometimes bypassing visual inputs.
We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom's Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0\% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria.