PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
This addresses the challenge of leveraging unlabeled video data for robots to understand and interact with environments, representing an incremental improvement over existing methods that rely on annotated data.
The authors tackled the problem of predicting future scene representations from unlabeled video data by proposing PlaySlot, an object-centric model that infers object representations and latent actions, enabling video prediction and planning. Their results show that PlaySlot outperforms baselines in video prediction and allows sample-efficient learning of robot behaviors from unlabeled demonstrations.
Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on videos and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available on https://play-slot.github.io/PlaySlot/.