End to End Lip Synchronization with a Temporal AutoEncoder
This addresses the challenge of lip synchronization for video editing and text-to-speech applications, representing a strong specific gain in this domain.
The paper tackles the problem of synchronizing lip movements in video with audio by finding an optimal alignment using a dual-domain recurrent neural network trained on synthetic data, and it greatly outperforms existing methods on various benchmarks.
We study the problem of syncing the lip movement in a video with the audio stream. Our solution finds an optimal alignment using a dual-domain recurrent neural network that is trained on synthetic data we generate by dropping and duplicating video frames. Once the alignment is found, we modify the video in order to sync the two sources. Our method is shown to greatly outperform the literature methods on a variety of existing and new benchmarks. As an application, we demonstrate our ability to robustly align text-to-speech generated audio with an existing video stream. Our code and samples are available at https://github.com/itsyoavshalev/End-to-End-Lip-Synchronization-with-a-Temporal-AutoEncoder.