Audio Input Generates Continuous Frames to Synthesize Facial Video Using Generative Adiversarial Networks
This addresses the challenge of synthesizing realistic speech videos for applications like virtual avatars or video editing, though it appears incremental as it builds on existing GAN and GRU techniques.
The paper tackles the problem of generating facial videos from audio input by proposing a GAN-based method with a Convolutional GRU for temporal coherence, achieving relatively realistic output results from short audio clips.
This paper presents a simple method for speech videos generation based on audio: given a piece of audio, we can generate a video of the target face speaking this audio. We propose Generative Adversarial Networks (GAN) with cut speech audio input as condition and use Convolutional Gate Recurrent Unit (GRU) in generator and discriminator. Our model is trained by exploiting the short audio and the frames in this duration. For training, we cut the audio and extract the face in the corresponding frames. We designed a simple encoder and compare the generated frames using GAN with and without GRU. We use GRU for temporally coherent frames and the results show that short audio can produce relatively realistic output results.