Reference-Free Assessment of Physical Consistency in World Model-based Video Generation
For researchers using video generation for robotic simulation, this provides a practical evaluation method that reduces the need for costly human voting or ground-truth references.
The paper introduces reference-free measures for evaluating physical consistency in generated videos, using DROID-SLAM and SEA-RAFT to quantify inconsistencies. Filtering videos with their relative consistency assessment improves task success rates by over 8%, narrowing the simulation-to-reality gap.
We introduce reference-free measures for evaluating the physical consistency of generated videos, combining relative and absolute approaches to assess fidelity. Although tools like WorldGym or WorldEval enable robotic simulation via video generation, physical fidelity gaps often prevent these environments from accurately reproducing real-world task success rates of VLA models. Unlike existing evaluation methods, which require costly human voting (Elo) or unavailable ground-truth references (FVD), our approach utilizes DROID-SLAM and SEA-RAFT to quantify physical inconsistencies, motivated by WorldScore. Videos filtered using our relative consistency assessment show an improvement in task success rates of over 8%, effectively narrowing the simulation-to-reality gap. Furthermore, our absolute assessment enables spatio-temporal localization, providing visualization of when and where physical artifacts occur.