Pipeline for 3D reconstruction of the human body from AR/VR headset mounted egocentric cameras
This addresses the challenge of occluded and skewed body views in AR/VR systems, enabling mobile human telepresence, though it appears incremental as it builds on existing GAN and reconstruction methods.
The paper tackles the problem of 3D reconstruction of the full human body from egocentric viewpoints, such as AR/VR headset cameras, by proposing a pipeline that uses conditional GANs to translate skewed views to third-person views and then generates a 3D mesh with realistic proportions and rigging for applications like animation.
In this paper, we propose a novel pipeline for the 3D reconstruction of the full body from egocentric viewpoints. 3-D reconstruction of the human body from egocentric viewpoints is a challenging task as the view is skewed and the body parts farther from the cameras are occluded. One such example is the view from cameras installed below VR headsets. To achieve this task, we first make use of conditional GANs to translate the egocentric views to full body third-person views. This increases the comprehensibility of the image and caters to occlusions. The generated third-person view is further sent through the 3D reconstruction module that generates a 3D mesh of the body. We also train a network that can take the third person full-body view of the subject and generate the texture maps for applying on the mesh. The generated mesh has fairly realistic body proportions and is fully rigged allowing for further applications such as real-time animation and pose transfer in games. This approach can be key to a new domain of mobile human telepresence.