Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models
This addresses the problem of physical reasoning limitations in VLMs for AI and robotics applications, offering a modular and scalable solution that is incremental in its approach.
The paper tackles the challenge of physical reasoning in Vision-Language Models (VLMs) by introducing Physics Context Builders (PCBs), a modular framework that uses specialized smaller VLMs to generate physical scene descriptions, resulting in up to a 13.8% accuracy improvement on complex tasks and strong Sim2Real transfer.
Physical reasoning remains a significant challenge for Vision-Language Models (VLMs). This limitation arises from an inability to translate learned knowledge into predictions about physical behavior. Although continual fine-tuning can mitigate this issue, it is expensive for large models and impractical to perform repeatedly for every task. This necessitates the creation of modular and scalable ways to teach VLMs about physical reasoning. To that end, we introduce Physics Context Builders (PCBs), a modular framework where specialized smaller VLMs are fine-tuned to generate detailed physical scene descriptions. These can be used as physical contexts to enhance the reasoning capabilities of larger VLMs. PCBs enable the separation of visual perception from reasoning, allowing us to analyze their relative contributions to physical understanding. We perform experiments on CLEVRER and on Falling Tower, a stability detection dataset with both simulated and real-world scenes, to demonstrate that PCBs provide substantial performance improvements, increasing average accuracy by up to 13.8% on complex physical reasoning tasks. Notably, PCBs also show strong Sim2Real transfer, successfully generalizing from simulated training data to real-world scenes.