PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model
For researchers working on physical reasoning in VLMs, this work provides a novel agentic framework that improves performance on dynamic real-world scenarios.
PhysNote addresses spatio-temporal identity drift and volatility of inference-time insights in VLMs for dynamic physical reasoning, achieving 56.68% accuracy on PhysBench, a 4.96% improvement over the best multi-agent baseline.
Vision-Language Models (VLMs) have demonstrated strong performance on textbook-style physics problems, yet they frequently fail when confronted with dynamic real-world scenarios that require temporal consistency and causal reasoning across frames. We identify two fundamental challenges underlying these failures: (1) spatio-temporal identity drift, where objects lose their physical identity across successive frames and break causal chains, and (2) volatility of inference-time insights, where a model may occasionally produce correct physical reasoning but never consolidates it for future reuse. To address these challenges, we propose PhysNote, an agentic framework that enables VLMs to externalize and refine physical knowledge through self-generated "Knowledge Notes." PhysNote stabilizes dynamic perception through spatio-temporal canonicalization, organizes self-generated insights into a hierarchical knowledge repository, and drives an iterative reasoning loop that grounds hypotheses in visual evidence before consolidating verified knowledge. Experiments on PhysBench demonstrate that PhysNote achieves 56.68% overall accuracy, a 4.96% improvement over the best multi-agent baseline, with consistent gains across all four physical reasoning domains.