OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
This addresses the challenge of flexible humanoid control for robotics applications, though it appears incremental as it builds on existing teleoperation and learning methods.
The paper tackles the problem of whole-body humanoid teleoperation and autonomy by developing OmniH2O, a learning-based system that enables versatile control through VR, verbal instruction, or RGB camera, and demonstrates dexterity in real-world tasks like sports and object manipulation.
We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.