A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
This addresses the need for systematic assessment of AI model performance on fundamental concepts, though it is incremental as it builds on existing validation methods.
The paper tackles the problem of validating AI models' understanding of basic concepts like object recognition and positions by introducing the Scrapbook framework to generate datasets with many questions and linguistic variations, finding that models like MobileVLM-V2 struggle with positional information and show biases, indicating areas for improvement.
In this paper, we present the Scrapbook framework, a novel methodology designed to generate extensive datasets for probing the learned concepts of artificial intelligence (AI) models. The framework focuses on fundamental concepts such as object recognition, absolute and relative positions, and attribute identification. By generating datasets with a large number of questions about individual concepts and a wide linguistic variation, the Scrapbook framework aims to validate the model's understanding of these basic elements before tackling more complex tasks. Our experimental findings reveal that, while contemporary models demonstrate proficiency in recognizing and enumerating objects, they encounter challenges in comprehending positional information and addressing inquiries with additional constraints. Specifically, the MobileVLM-V2 model showed significant answer disagreements and plausible wrong answers, while other models exhibited a bias toward affirmative answers and struggled with questions involving geometric shapes and positional information, indicating areas for improvement in understanding and consistency. The proposed framework offers a valuable instrument for generating diverse and comprehensive datasets, which can be utilized to systematically assess and enhance the performance of AI models.