Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis
This work addresses the challenge of generating interactive and realistic virtual human performances for 3D applications, representing an incremental improvement by leveraging existing SMPL models.
The paper tackles the problem of creating realistic real-time animations of virtual humans by learning pose-dependent appearance and geometry from dynamic mesh sequences, achieving high realism and enabling streamlined real-time rendering.
We propose a novel representation of virtual humans for highly realistic real-time animation and rendering in 3D applications. We learn pose dependent appearance and geometry from highly accurate dynamic mesh sequences obtained from state-of-the-art multiview-video reconstruction. Learning pose-dependent appearance and geometry from mesh sequences poses significant challenges, as it requires the network to learn the intricate shape and articulated motion of a human body. However, statistical body models like SMPL provide valuable a-priori knowledge which we leverage in order to constrain the dimension of the search space enabling more efficient and targeted learning and define pose-dependency. Instead of directly learning absolute pose-dependent geometry, we learn the difference between the observed geometry and the fitted SMPL model. This allows us to encode both pose-dependent appearance and geometry in the consistent UV space of the SMPL model. This approach not only ensures a high level of realism but also facilitates streamlined processing and rendering of virtual humans in real-time scenarios.