Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
This work enables arbitrary-subject talking face generation, which is useful for applications like lip reading and audio-video retrieval, representing a novel method for a known bottleneck.
The paper tackles the problem of generating realistic talking face sequences from speech by disentangling subject-related and speech-related information, achieving clearer lip motion patterns on arbitrary subjects compared to previous work.
Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.