MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion
This work addresses the challenge of realistic human image retargeting for applications in animation, virtual reality, and content creation, representing an incremental improvement with a novel diffusion-based method.
The authors tackled the problem of generating new images of a person with controlled poses and facial expressions while preserving identity, achieving robust appearance control and generalization to unseen identities and complex poses without fine-tuning.
In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of (1) the pre-training of an appearance-control block and (2) learning appearance-disentangled pose control. Our novel design enables robust appearance control over generated human images, including body, facial attributes, and even background. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses without the need for additional fine-tuning. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. The code is available at: https://github.com/Boese0601/MagicDance