Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architectures
This addresses the problem of generating videos from data or text for applications in multimedia and AI, representing a novel approach as the first of its kind for text-to-video generation.
The paper tackles the problem of automatic video generation by introducing Sync-DRAW, a method that combines a Variational Autoencoder with a Recurrent Attention Mechanism to generate temporally dependent video frames, and it can also perform text-to-video generation, achieving high structural integrity in generated frames on datasets like Bouncing MNIST, KTH, and UCF-101.
This paper introduces a novel approach for generating videos called Synchronized Deep Recurrent Attentive Writer (Sync-DRAW). Sync-DRAW can also perform text-to-video generation which, to the best of our knowledge, makes it the first approach of its kind. It combines a Variational Autoencoder~(VAE) with a Recurrent Attention Mechanism in a novel manner to create a temporally dependent sequence of frames that are gradually formed over time. The recurrent attention mechanism in Sync-DRAW attends to each individual frame of the video in sychronization, while the VAE learns a latent distribution for the entire video at the global level. Our experiments with Bouncing MNIST, KTH and UCF-101 suggest that Sync-DRAW is efficient in learning the spatial and temporal information of the videos and generates frames with high structural integrity, and can generate videos from simple captions on these datasets. (Accepted as oral paper in ACM-Multimedia 2017)