HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions
This addresses motion capture limitations for applications requiring robust tracking of challenging human movements, though it is incremental as it builds on existing hybrid mocap methods.
The paper tackles the problem of monocular 3D motion capture failing with occlusions and complex movements by augmenting a single camera with 4 IMUs in a learning-and-optimization framework, achieving real-time performance up to 60 fps with state-of-the-art accuracy.
Monocular 3D motion capture (mocap) is beneficial to many applications. The use of a single camera, however, often fails to handle occlusions of different body parts and hence it is limited to capture relatively simple movements. We present a light-weight, hybrid mocap technique called HybridCap that augments the camera with only 4 Inertial Measurement Units (IMUs) in a learning-and-optimization framework. We first employ a weakly-supervised and hierarchical motion inference module based on cooperative Gated Recurrent Unit (GRU) blocks that serve as limb, body and root trackers as well as an inverse kinematics solver. Our network effectively narrows the search space of plausible motions via coarse-to-fine pose estimation and manages to tackle challenging movements with high efficiency. We further develop a hybrid optimization scheme that combines inertial feedback and visual cues to improve tracking accuracy. Extensive experiments on various datasets demonstrate HybridCap can robustly handle challenging movements ranging from fitness actions to Latin dance. It also achieves real-time performance up to 60 fps with state-of-the-art accuracy.