On the Benefits of Instance Decomposition in Video Prediction Models
This addresses a challenge in video prediction for intelligent agents like robots and autonomous vehicles, but it is incremental as it builds on existing latent-transformer models.
The paper tackles the problem of video prediction by explicitly modeling objects separately in dynamic scenes, showing that this decomposition leads to higher quality predictions compared to joint modeling approaches.
Video prediction is a crucial task for intelligent agents such as robots and autonomous vehicles, since it enables them to anticipate and act early on time-critical incidents. State-of-the-art video prediction methods typically model the dynamics of a scene jointly and implicitly, without any explicit decomposition into separate objects. This is challenging and potentially sub-optimal, as every object in a dynamic scene has their own pattern of movement, typically somewhat independent of others. In this paper, we investigate the benefit of explicitly modeling the objects in a dynamic scene separately within the context of latent-transformer video prediction models. We conduct detailed and carefully-controlled experiments on both synthetic and real-world datasets; our results show that decomposing a dynamic scene leads to higher quality predictions compared with models of a similar capacity that lack such decomposition.