Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild
This work addresses the problem of generating realistic talking faces from audio for applications in video synthesis, though it is incremental by introducing attention to an existing lip-syncing strategy.
The paper tackles the challenge of achieving accurate lip synchronization in talking face generation by proposing the AttnWav2Lip model, which incorporates spatial and channel attention mechanisms to focus on lip region reconstruction, demonstrating superior performance on benchmark datasets like LRW, LRS2, and LRS3 compared to baselines.
Talking face generation with great practical significance has attracted more attention in recent audio-visual studies. How to achieve accurate lip synchronization is a long-standing challenge to be further investigated. Motivated by xxx, in this paper, an AttnWav2Lip model is proposed by incorporating spatial attention module and channel attention module into lip-syncing strategy. Rather than focusing on the unimportant regions of the face image, the proposed AttnWav2Lip model is able to pay more attention on the lip region reconstruction. To our limited knowledge, this is the first attempt to introduce attention mechanism to the scheme of talking face generation. An extensive experiments have been conducted to evaluate the effectiveness of the proposed model. Compared to the baseline measured by LSE-D and LSE-C metrics, a superior performance has been demonstrated on the benchmark lip synthesis datasets, including LRW, LRS2 and LRS3.