Learning to Act without Actions
This addresses the challenge of leveraging abundant web videos for reinforcement learning, offering a novel approach to pre-training generalist policies, though it is an incremental step in the field.
The paper tackles the problem of pre-training reinforcement learning models from videos without action labels by introducing Latent Action Policies (LAPO), which recovers latent actions and enables training policies that achieve expert-level performance after fine-tuning.
Pre-training large models on vast amounts of web data has proven to be an effective approach for obtaining powerful, general models in domains such as language and vision. However, this paradigm has not yet taken hold in reinforcement learning. This is because videos, the most abundant form of embodied behavioral data on the web, lack the action labels required by existing methods for imitating behavior from demonstrations. We introduce Latent Action Policies (LAPO), a method for recovering latent action information, and thereby latent-action policies, world models, and inverse dynamics models, purely from videos. LAPO is the first method able to recover the structure of the true action space just from observed dynamics, even in challenging procedurally-generated environments. LAPO enables training latent-action policies that can be rapidly fine-tuned into expert-level policies, either offline using a small action-labeled dataset, or online with rewards. LAPO takes a first step towards pre-training powerful, generalist policies and world models on the vast amounts of videos readily available on the web.