The Power of Next-Frame Prediction for Learning Physical Laws
This work addresses the challenge of inducing visual understanding without explicit labels, offering a general learning strategy for AI systems, though it is incremental as it builds on existing methods like causal language modeling.
The paper tackled the problem of understanding visual dynamics by exploring next-frame prediction as a foundational learning strategy, and found that models trained this way could predict physical constants like gravity from simulation videos, improving loss by factors of 1.28 to 6.24 compared to random models.
Next-frame prediction is a useful and powerful method for modelling and understanding the dynamics of video data. Inspired by the empirical success of causal language modelling and next-token prediction in language modelling, we explore the extent to which next-frame prediction serves as a strong foundational learning strategy (analogous to language modelling) for inducing an understanding of the visual world. In order to quantify the specific visual understanding induced by next-frame prediction, we introduce six diagnostic simulation video datasets derived from fundamental physical laws created by varying physical constants such as gravity and mass. We demonstrate that our models trained only on next-frame prediction are capable of predicting the value of these physical constants (e.g. gravity) without having been trained directly to learn these constants via a regression task. We find that the generative training phase alone induces a model state that can predict physical constants significantly better than that of a random model, improving the loss by a factor of between 1.28 to 6.24. We conclude that next-frame prediction shows great promise as a general learning strategy to induce understanding of the many `laws' that govern the visual domain without the need for explicit labelling.