MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions
This work addresses the challenge of creating high-fidelity and multimodal video portraits for applications in animation and virtual communication, representing an incremental improvement over existing methods.
The paper tackles the problem of generating natural and realistic talking portrait videos from audio by addressing the lack of correlation between lip-sync and other movements like head pose and eye blinking, resulting in improved performance over previous methods as shown in extensive evaluations.
Audio-driven portrait animation aims to synthesize portrait videos that are conditioned by given audio. Animating high-fidelity and multimodal video portraits has a variety of applications. Previous methods have attempted to capture different motion modes and generate high-fidelity portrait videos by training different models or sampling signals from given videos. However, lacking correlation learning between lip-sync and other movements (e.g., head pose/eye blinking) usually leads to unnatural results. In this paper, we propose a unified system for multi-person, diverse, and high-fidelity talking portrait generation. Our method contains three stages, i.e., 1) Mapping-Once network with Dual Attentions (MODA) generates talking representation from given audio. In MODA, we design a dual-attention module to encode accurate mouth movements and diverse modalities. 2) Facial composer network generates dense and detailed face landmarks, and 3) temporal-guided renderer syntheses stable videos. Extensive evaluations demonstrate that the proposed system produces more natural and realistic video portraits compared to previous methods.