Olaf-World: Orienting Latent Actions for Video World Modeling
This work addresses a bottleneck in video world modeling for AI and robotics by enabling better transfer of learned actions without labeled data, though it is incremental as it builds on existing latent action learning methods.
The paper tackles the problem of scaling action-controllable world models by addressing the lack of action labels and poor transferability of learned latent actions across contexts, achieving stronger zero-shot action transfer and more data-efficient adaptation compared to state-of-the-art baselines.
Scaling action-controllable world models is limited by the scarcity of action labels. While latent action learning promises to extract control interfaces from unlabeled video, learned latents often fail to transfer across contexts: they entangle scene-specific cues and lack a shared coordinate system. This occurs because standard objectives operate only within each clip, providing no mechanism to align action semantics across contexts. Our key insight is that although actions are unobserved, their semantic effects are observable and can serve as a shared reference. We introduce Seq$Δ$-REPA, a sequence-level control-effect alignment objective that anchors integrated latent action to temporal feature differences from a frozen, self-supervised video encoder. Building on this, we present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video. Extensive experiments demonstrate that our method learns a more structured latent action space, leading to stronger zero-shot action transfer and more data-efficient adaptation to new control interfaces than state-of-the-art baselines.