Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction
This addresses the problem of scalable motion reconstruction for robotics and AI systems by providing a method to learn from diverse real-world interactions, though it relies on human annotations for contact points, indicating an incremental advance.
The paper tackles the challenge of extracting 4D human-object interaction data from monocular internet videos by introducing 4DHOISolver, an optimization framework that uses sparse human-in-the-loop contact annotations to achieve accurate and scalable reconstructions, resulting in the Open4DHOI dataset with 144 object types and 103 actions.
Generalized robots must learn from diverse, large-scale human-object interactions (HOI) to operate robustly in the real world. Monocular internet videos offer a nearly limitless and readily available source of data, capturing an unparalleled diversity of human activities, objects, and environments. However, accurately and scalably extracting 4D interaction data from these in-the-wild videos remains a significant and unsolved challenge. Thus, in this work, we introduce 4DHOISolver, a novel and efficient optimization framework that constrains the ill-posed 4D HOI reconstruction problem by leveraging sparse, human-in-the-loop contact point annotations, while maintaining high spatio-temporal coherence and physical plausibility. Leveraging this framework, we introduce Open4DHOI, a new large-scale 4D HOI dataset featuring a diverse catalog of 144 object types and 103 actions. Furthermore, we demonstrate the effectiveness of our reconstructions by enabling an RL-based agent to imitate the recovered motions. However, a comprehensive benchmark of existing 3D foundation models indicates that automatically predicting precise human-object contact correspondences remains an unsolved problem, underscoring the immediate necessity of our human-in-the-loop strategy while posing an open challenge to the community. Data and code will be publicly available at https://wenboran2002.github.io/open4dhoi/