From Here to There: Video Inbetweening Using Direct 3D Convolutions
This addresses video generation for applications like animation or simulation, but it is incremental as it builds on existing stochastic video generation methods.
The paper tackles the problem of generating plausible and diverse video sequences between given start and end frames, known as inbetweening, by proposing a fully convolutional model that uses direct 3D convolutions and adversarial training, achieving meaningful results on benchmark datasets.
We consider the problem of generating plausible and diverse video sequences, when we are only given a start and an end frame. This task is also known as inbetweening, and it belongs to the broader area of stochastic video generation, which is generally approached by means of recurrent neural networks (RNN). In this paper, we propose instead a fully convolutional model to generate video sequences directly in the pixel domain. We first obtain a latent video representation using a stochastic fusion mechanism that learns how to incorporate information from the start and end frames. Our model learns to produce such latent representation by progressively increasing the temporal resolution, and then decode in the spatiotemporal domain using 3D convolutions. The model is trained end-to-end by minimizing an adversarial loss. Experiments on several widely-used benchmark datasets show that it is able to generate meaningful and diverse in-between video sequences, according to both quantitative and qualitative evaluations.