Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation
This addresses the problem of high-quality talking head video generation for applications like virtual avatars and video editing, though it appears incremental as it builds on existing codebook and transformer techniques.
The paper tackles the challenge of generating realistic talking head videos that simultaneously preserve accurate poses and fine-grained facial details by proposing a unified framework that learns multi-scale motion and appearance codebooks and uses a transformer-based compensation module. The approach demonstrates superior qualitative and quantitative results compared to state-of-the-art methods on various benchmarks.
Talking head video generation aims to generate a realistic talking head video that preserves the person's identity from a source image and the motion from a driving video. Despite the promising progress made in the field, it remains a challenging and critical problem to generate videos with accurate poses and fine-grained facial details simultaneously. Essentially, facial motion is often highly complex to model precisely, and the one-shot source face image cannot provide sufficient appearance guidance during generation due to dynamic pose changes. To tackle the problem, we propose to jointly learn motion and appearance codebooks and perform multi-scale codebook compensation to effectively refine both the facial motion conditions and appearance features for talking face image decoding. Specifically, the designed multi-scale motion and appearance codebooks are learned simultaneously in a unified framework to store representative global facial motion flow and appearance patterns. Then, we present a novel multi-scale motion and appearance compensation module, which utilizes a transformer-based codebook retrieval strategy to query complementary information from the two codebooks for joint motion and appearance compensation. The entire process produces motion flows of greater flexibility and appearance features with fewer distortions across different scales, resulting in a high-quality talking head video generation framework. Extensive experiments on various benchmarks validate the effectiveness of our approach and demonstrate superior generation results from both qualitative and quantitative perspectives when compared to state-of-the-art competitors.