An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing
This work addresses the need for robust, real-time 3D applications like tele-immersion, but it appears incremental as it builds on existing volumetric Fourier transform methods.
The paper tackles the problem of real-time 3D human reconstruction and motion capturing by proposing an integrated platform with a novel calibration method, enhanced reconstruction algorithm, and evaluation framework, achieving validated results with multi-Kinect2 datasets.
The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeleton-based motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors' calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texture-mapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies.