DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare
This addresses the challenge of accurate 3D human body reconstruction from monocular images, which is important for applications in computer vision and graphics, and is an incremental improvement with a novel integration of render-and-compare.
The paper tackles the problem of jointly estimating 3D human pose and shape from a single RGB image, achieving superior performance against state-of-the-art methods on public benchmarks.
We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image. Our two-step framework takes the body pixel-to-surface correspondence map (i.e., IUV map) as proxy representation and then performs estimation of parameterized human pose and shape. Specifically, given an estimated IUV map, we develop a deep neural network optimizing 3D body reconstruction losses and further integrating a render-and-compare scheme to minimize differences between the input and the rendered output, i.e., dense body landmarks, body part masks, and adversarial priors. To boost learning, we further construct a large-scale synthetic dataset (MOCA) utilizing web-crawled Mocap sequences, 3D scans and animations. The generated data covers diversified camera views, human actions and body shapes, and is paired with full ground truth. Our model jointly learns to represent the 3D human body from hybrid datasets, mitigating the problem of unpaired training data. Our experiments show that DenseRaC obtains superior performance against state of the art on public benchmarks of various humanrelated tasks.