$Δ$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos
For computer vision and physics simulation, this work addresses the generalization gap of existing methods to complex real-world settings by leveraging language-based representation.
ΔYNAMICS uses language as a unified representation to infer rigid-body dynamics from monocular videos, achieving a 7x improvement in segmentation IoU (0.30) over leading VLMs on CLEVRER, with further gains via test-time sampling (27%) and evolutionary search (120%), and demonstrates strong transfer to real-world videos.
Inferring rigid-body physical states and properties from monocular videos is a fundamental step toward physics-based perception and simulation. Existing approaches assume specific underlying physical systems, object types, and camera poses, making them unable to generalize to complex real-world settings. We introduce $Δ$YNAMICS, a vision-language framework that uses language as a unified representation of rigid-body dynamics. Instead of directly predicting parameters, $Δ$YNAMICS generates scene configurations in a structured text format for physics simulation. We enhance the model's generalization by integrating natural language motion reasoning and leveraging optical flow as a semantic-agnostic input. On the CLEVRER dataset, $Δ$YNAMICS achieves a segmentation IoU of 0.30, a 7x improvement over leading VLMs (InternVL3-8B, Qwen2.5-VL-7B and Claude-4-Sonnet). Additionally, test-time sampling and evolutionary search further boost performance by 27% and 120% in segmentation IoU, respectively. Finally, we demonstrate strong transfer to a new dataset of 235 real-world rigid-body videos, highlighting the potential of language-driven physics inference for bridging perception and simulation.