Probabilistic Video Generation using Holistic Attribute Control
This work addresses the problem of generating diverse and consistent video sequences for applications in computer vision, but it appears incremental as it builds on existing VAEs and RNNs with attribute controls.
The paper tackles video generation and future prediction by proposing a generative framework that decodes samples from a latent space into video frames, using VAEs and RNNs to model dynamics and improve consistency with attribute controls. Experimental results on datasets like Chair CAD, Weizmann Human Action, and MIT-Flickr show effectiveness compared to state-of-the-art methods, though no concrete numbers are provided.
Videos express highly structured spatio-temporal patterns of visual data. A video can be thought of as being governed by two factors: (i) temporally invariant (e.g., person identity), or slowly varying (e.g., activity), attribute-induced appearance, encoding the persistent content of each frame, and (ii) an inter-frame motion or scene dynamics (e.g., encoding evolution of the person ex-ecuting the action). Based on this intuition, we propose a generative framework for video generation and future prediction. The proposed framework generates a video (short clip) by decoding samples sequentially drawn from a latent space distribution into full video frames. Variational Autoencoders (VAEs) are used as a means of encoding/decoding frames into/from the latent space and RNN as a wayto model the dynamics in the latent space. We improve the video generation consistency through temporally-conditional sampling and quality by structuring the latent space with attribute controls; ensuring that attributes can be both inferred and conditioned on during learning/generation. As a result, given attributes and/orthe first frame, our model is able to generate diverse but highly consistent sets ofvideo sequences, accounting for the inherent uncertainty in the prediction task. Experimental results on Chair CAD, Weizmann Human Action, and MIT-Flickr datasets, along with detailed comparison to the state-of-the-art, verify effectiveness of the framework.