Interpretable Physics Reasoning and Performance Taxonomy in Vision-Language Models
This work addresses the need for rigorous evaluation of scientific reasoning in VLMs, which is incremental as it builds on existing VLM capabilities by providing a new testbed.
The authors tackled the problem of evaluating vision-language models' understanding of fundamental physics principles by introducing a novel framework with over 400 problems across four domains, finding that model scale correlates with reasoning ability, with the top model achieving an overall score of 0.815.
As Vision-Language Models (VLMs) grow in sophistication, their ability to perform reasoning is coming under increasing supervision. While they excel at many tasks, their grasp of fundamental scientific principles, such as physics, remains an underexplored frontier. To reflect the advancements in these capabilities, we introduce a novel and accessible framework designed to rigorously evaluate VLMs on their understanding of 2D physics. Our framework features a pragmatic scenario generator that creates a diverse testbed of over 400 problems across four core domains: Projectile Motion, Collision Dynamics, Mechanics, and Fluid Dynamics. Through comprehensive evaluation of four state-of-the-art VLMs, we demonstrate a strong correlation between model scale and reasoning ability, with our top-performing model, Qwen2.5-VL-7B, achieving an overall score of 0.815. We find that while models excel at formulaic problems, they struggle significantly with domains requiring abstract spatial reasoning. By designing this framework, we aim to democratize the study of scientific reasoning in VLMs and foster deeper insights into their capabilities and limitations.