Joint Learning of Depth and Appearance for Portrait Image Animation
This work addresses the under-explored challenge of co-generating consistent visual and 3D outputs for portrait animation, which is incremental as it builds on existing diffusion models.
The paper tackles the problem of generating consistent visual and 3D outputs for portrait animation by jointly learning appearance and depth in a diffusion-based generator, enabling applications like facial depth-to-image generation and audio-driven talking head animation.
2D portrait animation has experienced significant advancements in recent years. Much research has utilized the prior knowledge embedded in large generative diffusion models to enhance high-quality image manipulation. However, most methods only focus on generating RGB images as output, and the co-generation of consistent visual plus 3D output remains largely under-explored. In our work, we propose to jointly learn the visual appearance and depth simultaneously in a diffusion-based portrait image generator. Our method embraces the end-to-end diffusion paradigm and introduces a new architecture suitable for learning this conditional joint distribution, consisting of a reference network and a channel-expanded diffusion backbone. Once trained, our framework can be efficiently adapted to various downstream applications, such as facial depth-to-image and image-to-depth generation, portrait relighting, and audio-driven talking head animation with consistent 3D output.