Decoupling the components of geometric understanding in Vision Language Models
This work addresses the problem of assessing geometric understanding in VLMs for AI researchers, highlighting differences between human and machine learning origins, but it is incremental as it builds on existing cognitive science paradigms.
The study evaluated whether state-of-the-art vision language models (VLMs) can understand simple geometric concepts, finding that they consistently underperform human adults from the USA and an Amazonian indigenous group, with performance being more brittle and less robust in tasks requiring mental rotation.
Understanding geometry relies heavily on vision. In this work, we evaluate whether state-of-the-art vision language models (VLMs) can understand simple geometric concepts. We use a paradigm from cognitive science that isolates visual understanding of simple geometry from the many other capabilities it is often conflated with such as reasoning and world knowledge. We compare model performance with human adults from the USA, as well as with prior research on human adults without formal education from an Amazonian indigenous group. We find that VLMs consistently underperform both groups of human adults, although they succeed with some concepts more than others. We also find that VLM geometric understanding is more brittle than human understanding, and is not robust when tasks require mental rotation. This work highlights interesting differences in the origin of geometric understanding in humans and machines -- e.g. from printed materials used in formal education vs. interactions with the physical world or a combination of the two -- and a small step toward understanding these differences.