Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videos
This work addresses the problem of efficient and fast human reconstruction for applications like animation or VR, though it is incremental as it builds on existing mesh-based representations.
The paper tackles human reconstruction from RGB videos by optimizing a SMPL+D mesh and multi-resolution texture using only RGB images, silhouettes, and keypoints, achieving up to 24x faster training and 192x faster inference while capturing geometric details and reducing artifacts compared to NeRF-based methods.
Human reconstruction and synthesis from monocular RGB videos is a challenging problem due to clothing, occlusion, texture discontinuities and sharpness, and framespecific pose changes. Many methods employ deferred rendering, NeRFs and implicit methods to represent clothed humans, on the premise that mesh-based representations cannot capture complex clothing and textures from RGB, silhouettes, and keypoints alone. We provide a counter viewpoint to this fundamental premise by optimizing a SMPL+D mesh and an efficient, multi-resolution texture representation using only RGB images, binary silhouettes and sparse 2D keypoints. Experimental results demonstrate that our approach is more capable of capturing geometric details compared to visual hull, mesh-based methods. We show competitive novel view synthesis and improvements in novel pose synthesis compared to NeRF-based methods, which introduce noticeable, unwanted artifacts. By restricting the solution space to the SMPL+D model combined with differentiable rendering, we obtain dramatic speedups in compute, training times (up to 24x) and inference times (up to 192x). Our method therefore can be used as is or as a fast initialization to NeRF-based methods.