Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizer
This addresses the problem of creating realistic 4D head avatars for applications like virtual reality or gaming, offering a novel data-driven approach that improves synthesis quality.
The paper tackles 4D head avatar synthesis by using pseudo multi-view videos to avoid reliance on inaccurate 3DMM reconstruction, resulting in superior performance in reconstruction fidelity, geometry consistency, and motion control accuracy compared to previous methods.
In this paper, we propose a novel learning approach for feed-forward one-shot 4D head avatar synthesis. Different from existing methods that often learn from reconstructing monocular videos guided by 3DMM, we employ pseudo multi-view videos to learn a 4D head synthesizer in a data-driven manner, avoiding reliance on inaccurate 3DMM reconstruction that could be detrimental to the synthesis performance. The key idea is to first learn a 3D head synthesizer using synthetic multi-view images to convert monocular real videos into multi-view ones, and then utilize the pseudo multi-view videos to learn a 4D head synthesizer via cross-view self-reenactment. By leveraging a simple vision transformer backbone with motion-aware cross-attentions, our method exhibits superior performance compared to previous methods in terms of reconstruction fidelity, geometry consistency, and motion control accuracy. We hope our method offers novel insights into integrating 3D priors with 2D supervisions for improved 4D head avatar creation.