TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments
This addresses the need for accurate 3D human tracking in real-world scenarios with moving cameras, which is important for applications like robotics and AR/VR, though it appears incremental as it builds on existing HPS methods.
The paper tackles the problem of reliably estimating 3D human pose and shape in global coordinates from dynamic cameras, which is critical for applications but challenging due to entangled human and camera motion. TRACE achieves state-of-the-art performance on tracking and HPS benchmarks by using a novel 5D representation and architectural components.
Although the estimation of 3D human pose and shape (HPS) is rapidly progressing, current methods still cannot reliably estimate moving humans in global coordinates, which is critical for many applications. This is particularly challenging when the camera is also moving, entangling human and camera motion. To address these issues, we adopt a novel 5D representation (space, time, and identity) that enables end-to-end reasoning about people in scenes. Our method, called TRACE, introduces several novel architectural components. Most importantly, it uses two new "maps" to reason about the 3D trajectory of people over time in camera, and world, coordinates. An additional memory unit enables persistent tracking of people even during long occlusions. TRACE is the first one-stage method to jointly recover and track 3D humans in global coordinates from dynamic cameras. By training it end-to-end, and using full image information, TRACE achieves state-of-the-art performance on tracking and HPS benchmarks. The code and dataset are released for research purposes.