Which priors matter? Benchmarking models for learning latent dynamics
This work provides a sober evaluation for researchers in robotics and autonomous driving, showing that current physically inspired methods are incremental and do not offer major improvements.
The authors benchmarked models that incorporate physical priors for learning latent dynamics from visual observations, finding that these methods do not significantly outperform standard techniques, though continuous and time-reversible dynamics were beneficial across all models.
Learning dynamics is at the heart of many important applications of machine learning (ML), such as robotics and autonomous driving. In these settings, ML algorithms typically need to reason about a physical system using high dimensional observations, such as images, without access to the underlying state. Recently, several methods have proposed to integrate priors from classical mechanics into ML models to address the challenge of physical reasoning from images. In this work, we take a sober look at the current capabilities of these models. To this end, we introduce a suite consisting of 17 datasets with visual observations based on physical systems exhibiting a wide range of dynamics. We conduct a thorough and detailed comparison of the major classes of physically inspired methods alongside several strong baselines. While models that incorporate physical priors can often learn latent spaces with desirable properties, our results demonstrate that these methods fail to significantly improve upon standard techniques. Nonetheless, we find that the use of continuous and time-reversible dynamics benefits models of all classes.