NWT: Towards natural audio-to-video generation with representation learning
This work addresses the problem of generating natural and expressive videos from audio for applications like media synthesis, though it is incremental as it builds on existing representation learning methods.
The paper tackles audio-to-video generation by introducing NWT, a model that learns its own latent representations (Memcodes) without domain-specific assumptions, and it achieves higher Mean Opinion Scores in video naturalness, facial expressiveness, and lipsync quality compared to other approaches.
In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation we call Memcodes. Memcodes are straightforward to implement, require no additional loss terms, are stable to train compared with other approaches, and show evidence of interpretability. To predict on the Memcode space, we use an autoregressive encoder-decoder model conditioned on audio. Additionally, our model can control latent attributes in the generated video that are not annotated in the data. We train NWT on clips from HBO's Last Week Tonight with John Oliver. NWT consistently scores above other approaches in Mean Opinion Score (MOS) on tests of overall video naturalness, facial naturalness and expressiveness, and lipsync quality. This work sets a strong baseline for generalized audio-to-video synthesis. Samples are available at https://next-week-tonight.github.io/NWT/.