HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device
It addresses motion generation for applications like VR/AR, but is incremental as it builds on existing methods like diffusion models and SLAM.
The paper tackles the problem of generating realistic full-body human motion from a single head-mounted device with a camera and visual SLAM, achieving online inference with minimal latency of 0.17 seconds and scaling to over 200 hours of motion data in complex environments.
This paper investigates the generation of realistic full-body human motion using a single head-mounted device with an outward-facing color camera and the ability to perform visual SLAM. To address the ambiguity of this setup, we present HMD^2, a novel system that balances motion reconstruction and generation. From a reconstruction standpoint, it aims to maximally utilize the camera streams to produce both analytical and learned features, including head motion, SLAM point cloud, and image embeddings. On the generative front, HMD^2 employs a multi-modal conditional motion diffusion model with a Transformer backbone to maintain temporal coherence of generated motions, and utilizes autoregressive inpainting to facilitate online motion inference with minimal latency (0.17 seconds). We show that our system provides an effective and robust solution that scales to a diverse dataset of over 200 hours of motion in complex indoor and outdoor environments.