Neural Allocentric Intuitive Physics Prediction from Real Videos
This work addresses the gap in computer vision for intuitive physics prediction from real-world data, offering a method that reduces reliance on strong assumptions or annotated datasets, though it appears incremental by building on existing spatial representation concepts.
The paper tackles the problem of predicting future physical dynamics from real videos by proposing a framework that learns layered representations, converting real images to a synthetic domain and using an allocentric viewpoint transformer and a novel RNN architecture. Experimental results demonstrate consistent and accurate multi-frame predictions with adaptation to real images.
Humans are able to make rich predictions about the future dynamics of physical objects from a glance. On the other hand, most existing computer vision approaches require strong assumptions about the underlying system, ad-hoc modeling, or annotated datasets, to carry out even simple predictions. To tackle this gap, we propose a new perspective on the problem of learning intuitive physics that is inspired by the spatial memory representation of objects and spaces in human brains, in particular the co-existence of egocentric and allocentric spatial representations. We present a generic framework that learns a layered representation of the physical world, using a cascade of invertible modules. In this framework, real images are first converted to a synthetic domain representation that reduces complexity arising from lighting and texture. Then, an allocentric viewpoint transformer removes viewpoint complexity by projecting images to a canonical view. Finally, a novel Recurrent Latent Variation Network (RLVN) architecture learns the dynamics of the objects interacting with the environment and predicts future motion, leveraging the availability of unlimited synthetic simulations. Predicted frames are then projected back to the original camera view and translated back to the real world domain. Experimental results show the ability of the framework to consistently and accurately predict several frames in the future and the ability to adapt to real images.