JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing
This work addresses a specific challenge in audio-driven talking-face video editing, which is incremental as it builds on existing generation research.
The paper tackles the problem of achieving precise lip-audio synchronization and high visual quality in talking-face video generation by introducing JoyGen, a two-stage framework that integrates audio features with facial depth maps, resulting in superior performance as demonstrated experimentally.
Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method.