ObamaNet: Photo-realistic lip-sync from text
This addresses the problem of creating realistic talking-head videos for applications like virtual assistants or entertainment, but it is incremental as it builds on existing neural modules like Char2Wav and Pix2Pix.
The paper tackles generating audio and synchronized photo-realistic lip-sync videos from text, achieving this with a fully neural architecture that does not rely on traditional computer graphics methods.
We present ObamaNet, the first architecture that generates both audio and synchronized photo-realistic lip-sync videos from any new text. Contrary to other published lip-sync approaches, ours is only composed of fully trainable neural modules and does not rely on any traditional computer graphics methods. More precisely, we use three main modules: a text-to-speech network based on Char2Wav, a time-delayed LSTM to generate mouth-keypoints synced to the audio, and a network based on Pix2Pix to generate the video frames conditioned on the keypoints.