Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting
This exposes a fundamental empirical limitation in current VLMs, which is an incremental but important problem for AI researchers focusing on visual reasoning.
The paper introduced VLMCountBench to test Vision-Language Models (VLMs) on counting tasks, finding that while VLMs count reliably with single shape types, they fail substantially when multiple shapes are combined in compositional counting.
Vision-Language Models (VLMs) have become a central focus of today's AI community, owing to their impressive abilities gained from training on large-scale vision-language data from the Web. These models have demonstrated strong performance across diverse tasks, including image understanding, video understanding, complex visual reasoning, and embodied AI. Despite these noteworthy successes, a fundamental question remains: Can VLMs count objects correctly? In this paper, we introduce a simple yet effective benchmark, VLMCountBench, designed under a minimalist setting with only basic geometric shapes (e.g., triangles, circles) and their compositions, focusing exclusively on counting tasks without interference from other factors. We adopt strict independent variable control and systematically study the effects of simple properties such as color, size, and prompt refinement in a controlled ablation. Our empirical results reveal that while VLMs can count reliably when only one shape type is present, they exhibit substantial failures when multiple shape types are combined (i.e., compositional counting). This highlights a fundamental empirical limitation of current VLMs and motivates important directions for future research.