EgoExo-WM: Unlocking Exo Video for Ego World Models
For researchers in embodied AI and robotics, this work addresses the data scarcity bottleneck in egocentric world model training by leveraging widely available exocentric video, though the method is incremental as it builds on existing pose extraction and video translation techniques.
The paper tackles the problem of limited egocentric training data for world models by proposing a method to convert abundant exocentric video into egocentric video using structured body pose extraction and human kinematics priors. This approach significantly improves prediction quality and downstream planning performance, enabling the use of in-the-wild exocentric data for egocentric world model training.
Egocentric world models present a promising direction for enabling agents to predict and plan, but their performance is constrained by the limited availability of egocentric training data and its inherent partial observability of humans' physical actions. In contrast, exocentric video is abundant and reveals body poses well, but lacks direct alignment with an agent's action space -- and is not egocentric. We propose a method to bridge this gap by extracting structured body pose from exocentric video as a representation of action and transforming the exocentric video to egocentric video, informed by a human kinematics prior. This process unlocks the integration of in-the-wild exocentric data for egocentric world model training. We show that training whole-body action-conditioned egocentric world models with our converted data significantly improves both prediction quality and downstream planning performance, where we infer the sequence of body poses needed to achieve a visual goal state. Our approach paves the way to enlist arbitrary in-the-wild videos for building powerful egocentric world models, furthering applications in robot planning and augmented-reality guidance.