Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing
This addresses the challenge of creating realistic 3D models of clothed humans from monocular images, which is important for applications in virtual reality, gaming, and animation, representing a significant advance over prior work.
The paper tackles the problem of photorealistic 3D human reconstruction from a single RGB image by introducing PHORHUM, an end-to-end deep neural network that estimates detailed 3D geometry and unshaded surface color with scene illumination, achieving state-of-the-art results in both geometric and color reconstruction.
We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. Our pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination. Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts. Moreover, our method specifically addresses methodological and practical limitations of prior work in terms of representing geometry, albedo, and illumination effects, in an end-to-end model where factors can be effectively disentangled. In extensive experiments, we demonstrate the versatility and robustness of our approach. Our state-of-the-art results validate the method qualitatively and for different metrics, for both geometric and color reconstruction.