Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
This work addresses the data bottleneck in whole-body humanoid-object interaction for robotics, offering a zero-shot solution that bypasses retargeting complexity.
Imagine2Real tackles the scarcity of 3D data for humanoid-object interaction by using video generative priors in a zero-shot framework, achieving flexible, geometry-free interaction without retargeting. It demonstrates robust physical deployment within a motion capture system.
Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data. While video generative priors offer a promising alternative, existing methods suffer from \textit{Representation Misalignment} due to their reliance on geometric priors (e.g., explicit CAD models), and \textit{Retargeting Complexity} arising from intensive morphing and morphological mismatch. We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction. To resolve misalignment, we formulate robot and object motions as unified 4D point trajectories. To overcome retargeting complexity, our Keypoints Tracker tracks only sparse critical points (base, hands, and object), entirely bypassing the error-amplifying retargeting process. To maintain natural gaits despite these sparse signals, we utilize the latent space of a Behavior Foundation Model (BFM) as the tracker's search domain. Using a progressive training strategy, Imagine2Real learns robust behaviors with simple tracking rewards, enabling zero-shot physical deployment within a motion capture(mocap) system.