Video-to-Video Translation for Visual Speech Synthesis
This addresses the challenge of multi-domain video translation for visual speech generation, which is incremental as it builds on image-to-image translation methods.
The paper tackles the problem of video-to-video translation for visual speech synthesis, aiming to transform input videos of spoken words into output videos of different words, and achieves this with a vocabulary of 500 words using a novel character-based GAN architecture.
Despite remarkable success in image-to-image translation that celebrates the advancements of generative adversarial networks (GANs), very limited attempts are known for video domain translation. We study the task of video-to-video translation in the context of visual speech generation, where the goal is to transform an input video of any spoken word to an output video of a different word. This is a multi-domain translation, where each word forms a domain of videos uttering this word. Adaptation of the state-of-the-art image-to-image translation model (StarGAN) to this setting falls short with a large vocabulary size. Instead we propose to use character encodings of the words and design a novel character-based GANs architecture for video-to-video translation called Visual Speech GAN (ViSpGAN). We are the first to demonstrate video-to-video translation with a vocabulary of 500 words.