Physics-based Human Pose Estimation from a Single Moving RGB Camera
This addresses the challenge of accurate human pose tracking for applications like robotics or AR/VR in dynamic, real-world environments with moving cameras and uneven terrain, representing a domain-specific advancement.
The paper tackles the problem of physics-based human pose estimation from a moving RGB camera in non-flat scenes by introducing MoviCam, a non-synthetic dataset with ground-truth camera trajectories and 3D human motion, and PhysDynPose, a method that incorporates scene geometry and physical constraints. The result shows that state-of-the-art methods struggle in this setting, while their method robustly estimates poses in world coordinates.
Most monocular and physics-based human pose tracking methods, while achieving state-of-the-art results, suffer from artifacts when the scene does not have a strictly flat ground plane or when the camera is moving. Moreover, these methods are often evaluated on in-the-wild real world videos without ground-truth data or on synthetic datasets, which fail to model the real world light transport, camera motion, and pose-induced appearance and geometry changes. To tackle these two problems, we introduce MoviCam, the first non-synthetic dataset containing ground-truth camera trajectories of a dynamically moving monocular RGB camera, scene geometry, and 3D human motion with human-scene contact labels. Additionally, we propose PhysDynPose, a physics-based method that incorporates scene geometry and physical constraints for more accurate human motion tracking in case of camera motion and non-flat scenes. More precisely, we use a state-of-the-art kinematics estimator to obtain the human pose and a robust SLAM method to capture the dynamic camera trajectory, enabling the recovery of the human pose in the world frame. We then refine the kinematic pose estimate using our scene-aware physics optimizer. From our new benchmark, we found that even state-of-the-art methods struggle with this inherently challenging setting, i.e. a moving camera and non-planar environments, while our method robustly estimates both human and camera poses in world coordinates.