Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
This work addresses the reasoning limitations of MLLMs for researchers in AI and computer vision, highlighting a critical gap in their ability to handle abstract visual tasks.
The study investigated whether multimodal large language models (MLLMs) can solve Bongard Problems (BPs) for abstract visual reasoning, finding that while MLLMs achieved some success on real-world datasets, they struggled significantly with synthetic BPs, indicating general limitations in abstract visual reasoning.
Abstract visual reasoning (AVR) involves discovering shared concepts across images through analogy, akin to solving IQ test problems. Bongard Problems (BPs) remain a key challenge in AVR, requiring both visual reasoning and verbal description. We investigate whether multimodal large language models (MLLMs) can solve BPs by formulating a set of diverse MLLM-suited solution strategies and testing $4$ proprietary and $4$ open-access models on $3$ BP datasets featuring synthetic (classic BPs) and real-world (Bongard HOI and Bongard-OpenWorld) images. Despite some successes on real-world datasets, MLLMs struggle with synthetic BPs. To explore this gap, we introduce Bongard-RWR, a dataset representing synthetic BP concepts using real-world images. Our findings suggest that weak MLLM performance on classical BPs is not due to the domain specificity, but rather comes from their general AVR limitations. Code and dataset are available at: https://github.com/pavonism/bongard-rwr