Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
This work addresses the challenge of producing visually intelligible lip movements in talking face generation, which is important for applications like video conferencing and assistive technologies, though it is incremental by building on existing synchronization and quality improvements.
The paper tackles the problem of generating talking faces with intelligible lip movements by using a lip-reading expert to penalize incorrect generations, achieving over 38% Word Error Rate reduction on LRS2 and 27.8% accuracy on LRW datasets compared to prior methods.
Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. The previous studies revealed the importance of lip-speech synchronization and visual quality. Despite much progress, they hardly focus on the content of lip movements i.e., the visual intelligibility of the spoken words, which is an important aspect of generation quality. To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results. Moreover, to compensate for data scarcity, we train the lip-reading expert in an audio-visual self-supervised manner. With a lip-reading expert, we propose a novel contrastive learning to enhance lip-speech synchronization, and a transformer to encode audio synchronically with video, while considering global temporal dependency of audio. For evaluation, we propose a new strategy with two different lip-reading experts to measure intelligibility of the generated videos. Rigorous experiments show that our proposal is superior to other State-of-the-art (SOTA) methods, such as Wav2Lip, in reading intelligibility i.e., over 38% Word Error Rate (WER) on LRS2 dataset and 27.8% accuracy on LRW dataset. We also achieve the SOTA performance in lip-speech synchronization and comparable performances in visual quality.