Do As I Do: Transferring Human Motion and Appearance between Monocular Videos with Spatial and Temporal Constraints
This addresses the challenge of creating plausible virtual actors from real images for applications in computer vision and graphics, representing an incremental improvement over existing appearance transfer methods.
The paper tackles the problem of transferring human motion and appearance between monocular videos, synthesizing new videos of people in different contexts while preserving motion constraints like feet touching the floor. It outperforms state-of-the-art methods in visual quality and metrics such as SSIM and LPIPS.
Creating plausible virtual actors from images of real actors remains one of the key challenges in computer vision and computer graphics. Marker-less human motion estimation and shape modeling from images in the wild bring this challenge to the fore. Although the recent advances on view synthesis and image-to-image translation, currently available formulations are limited to transfer solely style and do not take into account the character's motion and shape, which are by nature intermingled to produce plausible human forms. In this paper, we propose a unifying formulation for transferring appearance and retargeting human motion from monocular videos that regards all these aspects. Our method synthesizes new videos of people in a different context where they were initially recorded. Differently from recent appearance transferring methods, our approach takes into account body shape, appearance, and motion constraints. The evaluation is performed with several experiments using publicly available real videos containing hard conditions. Our method is able to transfer both human motion and appearance outperforming state-of-the-art methods, while preserving specific features of the motion that must be maintained (e.g., feet touching the floor, hands touching a particular object) and holding the best visual quality and appearance metrics such as Structural Similarity (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS).