Improved Conditional VRNNs for Video Prediction
This work improves video prediction for applications like robotics and autonomous driving, but it is incremental as it builds on existing latent variable models.
The paper tackled the problem of blurry predictions in video frame prediction by addressing underfitting in variational autoencoders, resulting in favorable performance across several metrics on three datasets.
Predicting future frames for a video sequence is a challenging generative modeling task. Promising approaches include probabilistic latent variable models such as the Variational Auto-Encoder. While VAEs can handle uncertainty and model multiple possible future outcomes, they have a tendency to produce blurry predictions. In this work we argue that this is a sign of underfitting. To address this issue, we propose to increase the expressiveness of the latent distributions and to use higher capacity likelihood models. Our approach relies on a hierarchy of latent variables, which defines a family of flexible prior and posterior distributions in order to better model the probability of future sequences. We validate our proposal through a series of ablation experiments and compare our approach to current state-of-the-art latent variable models. Our method performs favorably under several metrics in three different datasets.