Neural Head Reenactment with Latent Pose Descriptors
This work addresses the challenge of realistic head reenactment for applications in video editing and virtual avatars, though it appears incremental as it builds on existing neural methods with a focus on latent representations.
The authors tackled the problem of neural head reenactment by developing a system that uses a latent pose representation learned solely from image reconstruction losses, enabling it to decompose pose from identity and perform cross-person reenactment while predicting foreground segmentation and RGB images.
We propose a neural head reenactment system, which is driven by a latent pose representation and is capable of predicting the foreground segmentation alongside the RGB image. The latent pose representation is learned as a part of the entire reenactment system, and the learning process is based solely on image reconstruction losses. We show that despite its simplicity, with a large and diverse enough training dataset, such learning successfully decomposes pose from identity. The resulting system can then reproduce mimics of the driving person and, furthermore, can perform cross-person reenactment. Additionally, we show that the learned descriptors are useful for other pose-related tasks, such as keypoint prediction and pose-based retrieval.