Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera
This addresses the challenge of physically plausible human rendering in computer vision and graphics, offering an incremental improvement by incorporating motion dynamics neglected in prior methods.
The paper tackled the problem of rendering dynamic humans with motion-dependent appearance from a single camera, achieving high-fidelity, temporally coherent videos for unseen poses and novel views by learning a compact equivariant representation from spatial and temporal derivatives.
Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i.e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved. Such appearance modeling conditioned on motion has been largely neglected in existing human rendering methods, resulting in rendering of physically implausible motion. A key challenge of learning the dynamics of the appearance lies in the requirement of a prohibitively large amount of observations. In this paper, we present a compact motion representation by enforcing equivariance -- a representation is expected to be transformed in the way that the pose is transformed. We model an equivariant encoder that can generate the generalizable representation from the spatial and temporal derivatives of the 3D body surface. This learned representation is decoded by a compositional multi-task decoder that renders high fidelity time-varying appearance. Our experiments show that our method can generate a temporally coherent video of dynamic humans for unseen body poses and novel views given a single view video.