Continual Predictive Learning from Videos
This addresses the challenge of building world models for physical processes in changing environments, which is incremental as it combines video prediction and continual learning techniques.
The paper tackles the problem of catastrophic forgetting in video prediction when tasks arrive sequentially in non-stationary environments, proposing a continual predictive learning approach that effectively mitigates forgetting and outperforms existing methods on new benchmarks based on RoboNet and KTH.
Predictive learning ideally builds the world model of physical processes in one or more given environments. Typical setups assume that we can collect data from all environments at all times. In practice, however, different prediction tasks may arrive sequentially so that the environments may change persistently throughout the training procedure. Can we develop predictive learning algorithms that can deal with more realistic, non-stationary physical environments? In this paper, we study a new continual learning problem in the context of video prediction, and observe that most existing methods suffer from severe catastrophic forgetting in this setup. To tackle this problem, we propose the continual predictive learning (CPL) approach, which learns a mixture world model via predictive experience replay and performs test-time adaptation with non-parametric task inference. We construct two new benchmarks based on RoboNet and KTH, in which different tasks correspond to different physical robotic environments or human actions. Our approach is shown to effectively mitigate forgetting and remarkably outperform the naïve combinations of previous art in video prediction and continual learning.