LumièreNet: Lecture Video Synthesis from Audio
This addresses the need for automated lecture video creation for educators and content creators, representing a novel method for a known bottleneck in video synthesis.
The authors tackled the problem of generating high-quality, full-pose headshot lecture videos from audio narration using LumièreNet, a deep-learning architecture that learns mapping functions from audio to video through pose-based latent codes, resulting in synthesized videos of any length.
We present LumièreNet, a simple, modular, and completely deep-learning based architecture that synthesizes, high quality, full-pose headshot lecture videos from instructor's new audio narration of any length. Unlike prior works, LumièreNet is entirely composed of trainable neural network modules to learn mapping functions from the audio to video through (intermediate) estimated pose-based compact and abstract latent codes. Our video demos are available at [22] and [23].