Vid2Sid: Videos Can Help Close the Sim2Real Gap
This work addresses the sim2real gap for robotics calibration, providing an interpretable alternative to black-box optimizers, though it is incremental in combining existing perception and optimization techniques.
The authors tackled the problem of calibrating robot simulator physics parameters to match real hardware by introducing Vid2Sid, a video-driven system identification pipeline that uses foundation-model perception and a VLM-in-the-loop optimizer to analyze paired sim-real videos, diagnose mismatches, and propose updates with natural language rationales. The result showed that Vid2Sid achieved the best average rank on sim2real holdout controls, matched or exceeded black-box optimizers, and recovered ground-truth parameters most accurately with a mean relative error under 13% versus 28-98% for other methods.
Calibrating a robot simulator's physics parameters (friction, damping, material stiffness) to match real hardware is often done by hand or with black-box optimizers that reduce error but cannot explain which physical discrepancies drive the error. When sensing is limited to external cameras, the problem is further compounded by perception noise and the absence of direct force or state measurements. We present Vid2Sid, a video-driven system identification pipeline that couples foundation-model perception with a VLM-in-the-loop optimizer that analyzes paired sim-real videos, diagnoses concrete mismatches, and proposes physics parameter updates with natural language rationales. We evaluate our approach on a tendon-actuated finger (rigid-body dynamics in MuJoCo) and a deformable continuum tentacle (soft-body dynamics in PyElastica). On sim2real holdout controls unseen during training, Vid2Sid achieves the best average rank across all settings, matching or exceeding black-box optimizers while uniquely providing interpretable reasoning at each iteration. Sim2sim validation confirms that Vid2Sid recovers ground-truth parameters most accurately (mean relative error under 13\% vs. 28--98\%), and ablation analysis reveals three calibration regimes. VLM-guided optimization excels when perception is clean and the simulator is expressive, while model-class limitations bound performance in more challenging settings.