The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?
This work addresses a gap in benchmarking MLLMs for simple perception, which is crucial for AI systems aiming to mimic human-like intelligence, though it is incremental in focusing on a specific evaluation aspect.
The paper tackles the problem of evaluating Multimodal Large Language Models (MLLMs) on basic visual perception skills, finding that state-of-the-art models perform weakly compared to high human performance on the Percept-V dataset, with performance declining as image complexity increases.
Cognitive science research treats visual perception, the ability to understand and make sense of a visual input, as one of the early developmental signs of intelligence. Its TVPS-4 framework categorizes and tests human perception into seven skills such as visual discrimination, and form constancy. Do Multimodal Large Language Models (MLLMs) match up to humans in basic perception? Even though there are many benchmarks that evaluate MLLMs on advanced reasoning and knowledge skills, there is limited research that focuses evaluation on simple perception. In response, we introduce Percept-V, a dataset containing 6000 program-generated uncontaminated images divided into 30 domains, where each domain tests one or more TVPS-4 skills. Our focus is on perception, so we make our domains quite simple and the reasoning and knowledge required for solving them are minimal. Since modern-day MLLMs can solve much more complex tasks, our a-priori expectation is that they will solve these domains very easily. Contrary to our belief, our experiments show a weak performance of SoTA proprietary and open-source MLLMs compared to very high human performance on Percept-V. We find that as number of objects in the image increases, performance goes down rather fast. Our experiments also identify the perception skills that are considerably harder for all models.