NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models
This provides a benchmark for assessing intelligence and critical thinking in models, addressing a gap in complex reasoning evaluation, but it is incremental as it focuses on dataset creation rather than novel methods.
The authors tackled the problem of evaluating cognitive multi-modal reasoning in vision-language models by introducing NTSEBench, a dataset of 2,728 multiple-choice questions with 4,642 images from the NTSE exam, and established baselines using state-of-the-art models.
Cognitive textual and visual reasoning tasks, including puzzles, series, and analogies, demand the ability to quickly reason, decipher, and evaluate patterns both textually and spatially. Due to extensive training on vast amounts of human-curated data, LLMs and VLMs excel in common-sense reasoning tasks, however still struggle with more complex reasoning that demands deeper cognitive understanding. We introduce NTSEBench, a new dataset designed to evaluate cognitive multi-modal reasoning and problem-solving skills of large models. The dataset contains 2728 multiple-choice questions, accompanied by a total of 4,642 images, categorized into 26 different types. These questions are drawn from the nationwide NTSE examination in India and feature a mix of visual and textual general aptitude challenges, designed to assess intelligence and critical thinking skills beyond mere rote learning. We establish baselines on the dataset using state-of-the-art LLMs and VLMs. To facilitate a comparison between open source and propriety models, we propose four distinct modeling strategies to handle different modalities -- text and images -- in the dataset instances.