Flow and Depth Assisted Video Prediction with Latent Transformer
This work addresses occlusion issues in video prediction for applications like robotics and world modeling, representing an incremental improvement.
The paper tackled the challenge of occlusion in video prediction by incorporating explicit motion and geometric structure information via point-flow and depth-maps, resulting in improved performance in occluded scenarios and more accurate background motion predictions compared to models without these modalities.
Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occlusion is still an inherent challenge in video prediction. We hypothesize that providing explicit information about motion (via point-flow) and geometric structure (via depth-maps) will enable video prediction models to perform better in situations with occlusion and the background motion. To investigate this, we present the first systematic study dedicated to occluded video prediction. We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow. We evaluate this model in a controlled setting on both synthetic and real-world datasets with not only appearance-based metrics but also Wasserstein distances on object masks, which can effectively measure the motion distribution of the prediction. We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.