Latent Action Pretraining Through World Modeling
This work addresses the problem of practical deployment for robotic manipulation models by enabling efficient, unsupervised pretraining, though it is incremental as it builds on existing latent action methods.
The paper tackles the challenge of deploying large Vision-Language-Action models in real-world robotics by proposing LAWM, a model-agnostic framework for self-supervised pretraining using latent action representations from unlabeled video data, which outperforms ground-truth action models and similar methods on benchmarks while being more efficient.
Vision-Language-Action (VLA) models have gained popularity for learning robotic manipulation tasks that follow language instructions. State-of-the-art VLAs, such as OpenVLA and $π_{0}$, were trained on large-scale, manually labeled action datasets collected through teleoperation. More recent approaches, including LAPA and villa-X, introduce latent action representations that enable unsupervised pretraining on unlabeled datasets by modeling abstract visual changes between frames. Although these methods have shown strong results, their large model sizes make deployment in real-world settings challenging. In this work, we propose LAWM, a model-agnostic framework to pretrain imitation learning models in a self-supervised way, by learning latent action representations from unlabeled video data through world modeling. These videos can be sourced from robot recordings or videos of humans performing actions with everyday objects. Our framework is designed to be effective for transferring across tasks, environments, and embodiments. It outperforms models trained with ground-truth robotics actions and similar pretraining methods on the LIBERO benchmark and real-world setup, while being significantly more efficient and practical for real-world settings.