Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
This work solves the underfitting issue in large-scale video prediction for applications like robotics, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of video prediction models underfitting on large, diverse datasets by introducing Greedy Hierarchical Variational Autoencoders (GHVAEs), which address memory constraints and optimization challenges through greedy, modular training. The result includes 17-55% gains in prediction performance on four datasets and a 35-40% higher success rate on real robot tasks.
A video prediction model that generalizes to diverse scenes would enable intelligent agents such as robots to perform a variety of tasks via planning with the model. However, while existing video prediction models have produced promising results on small datasets, they suffer from severe underfitting when trained on large and diverse datasets. To address this underfitting challenge, we first observe that the ability to train larger video prediction models is often bottlenecked by the memory constraints of GPUs or TPUs. In parallel, deep hierarchical latent variable models can produce higher quality predictions by capturing the multi-level stochasticity of future observations, but end-to-end optimization of such models is notably difficult. Our key insight is that greedy and modular optimization of hierarchical autoencoders can simultaneously address both the memory constraints and the optimization challenges of large-scale video prediction. We introduce Greedy Hierarchical Variational Autoencoders (GHVAEs), a method that learns high-fidelity video predictions by greedily training each level of a hierarchical autoencoder. In comparison to state-of-the-art models, GHVAEs provide 17-55% gains in prediction performance on four video datasets, a 35-40% higher success rate on real robot tasks, and can improve performance monotonically by simply adding more modules.