Generating Videos with Scene Dynamics
This work addresses the challenge of video understanding and simulation by leveraging unlabeled data, offering incremental improvements in generative modeling and representation learning for video applications.
The paper tackles the problem of learning scene dynamics from unlabeled video for both recognition and generation tasks, proposing a generative adversarial network that separates foreground from background and showing it can generate short videos and predict plausible futures from static images, with experiments indicating it learns useful features for action recognition with minimal supervision.
We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.