ViPro: Enabling and Controlling Video Prediction for Complex Dynamical Scenarios using Procedural Knowledge
This addresses the challenge of video prediction for scenarios where state-of-the-art methods fail, by incorporating prior knowledge to make learning feasible.
The paper tackles the problem of video prediction in complex dynamical scenarios by integrating procedural domain knowledge into data-driven models, resulting in a symbolically addressable interface that enables control tasks.
We propose a novel architecture design for video prediction in order to utilize procedural domain knowledge directly as part of the computational graph of data-driven models. On the basis of new challenging scenarios we show that state-of-the-art video predictors struggle in complex dynamical settings, and highlight that the introduction of prior process knowledge makes their learning problem feasible. Our approach results in the learning of a symbolically addressable interface between data-driven aspects in the model and our dedicated procedural knowledge module, which we utilize in downstream control tasks.