InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
This addresses the limitation of conventional video dubbing that only edits mouth regions, improving viewer immersion for applications in entertainment and communication.
The paper tackles the problem of generating full-body motion in video dubbing by introducing sparse-frame video dubbing, which preserves reference keyframes to maintain identity and gestures while enabling audio-synchronized editing, achieving state-of-the-art performance on datasets like HDTF, CelebV-HQ, and EMTD with superior visual realism and motion synchronization.
Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing, a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization.