Learning Skills from Action-Free Videos
This addresses the challenge of translating video priors into actionable robot skills, offering a step toward generalist robots, though it is incremental by building on existing latent-action and video generative models.
The paper tackles the problem of learning robot skills from action-free videos by introducing Skill Abstraction from Optical Flow (SOF), which learns latent skills through optical flow to bridge video dynamics and actions, resulting in improved performance in multitask and long-horizon settings.
Learning from videos offers a promising path toward generalist robots by providing rich visual and temporal priors beyond what real robot datasets contain. While existing video generative models produce impressive visual predictions, they are difficult to translate into low-level actions. Conversely, latent-action models better align videos with actions, but they typically operate at the single-step level and lack high-level planning capabilities. We bridge this gap by introducing Skill Abstraction from Optical Flow (SOF), a framework that learns latent skills from large collections of action-free videos. Our key idea is to learn a latent skill space through an intermediate representation based on optical flow that captures motion information aligned with both video dynamics and robot actions. By learning skills in this flow-based latent space, SOF enables high-level planning over video-derived skills and allows for easier translation of these skills into actions. Experiments show that our approach consistently improves performance in both multitask and long-horizon settings, demonstrating the ability to acquire and compose skills directly from raw visual data.