DualLip: A System for Joint Lip Reading and Generation
This work addresses the problem of limited labeled data for lip reading and generation, which is crucial for applications like talking face generation, though it is incremental in leveraging task duality and pseudo-labeling.
The paper tackles the dual tasks of lip reading and lip generation by developing DualLip, a system that leverages unlabeled text and lip video data to improve both tasks, achieving a 1.16% character error rate and 2.71% word error rate on the GRID benchmark for lip reading and surpassing performance with only 10% paired data for lip generation.
Lip reading aims to recognize text from talking lip, while lip generation aims to synthesize talking lip according to text, which is a key component in talking face generation and is a dual task of lip reading. In this paper, we develop DualLip, a system that jointly improves lip reading and generation by leveraging the task duality and using unlabeled text and lip video data. The key ideas of the DualLip include: 1) Generate lip video from unlabeled text with a lip generation model, and use the pseudo pairs to improve lip reading; 2) Generate text from unlabeled lip video with a lip reading model, and use the pseudo pairs to improve lip generation. We further extend DualLip to talking face generation with two additionally introduced components: lip to face generation and text to speech generation. Experiments on GRID and TCD-TIMIT demonstrate the effectiveness of DualLip on improving lip reading, lip generation, and talking face generation by utilizing unlabeled data. Specifically, the lip generation model in our DualLip system trained with only10% paired data surpasses the performance of that trained with the whole paired data. And on the GRID benchmark of lip reading, we achieve 1.16% character error rate and 2.71% word error rate, outperforming the state-of-the-art models using the same amount of paired data.