World Model Self-Distillation: Training World Models to Solve General Tasks
This work addresses the scalability bottleneck of collecting task-execution videos for training world models to solve tasks, offering a method that leverages unlabeled images and VLM-generated solutions.
The paper introduces a framework that combines self-distillation and reinforcement learning to enable pretrained video generators to solve general tasks without requiring paired task-video data. The Executor model outperforms the Demonstrator on the WorldTasks-Benchmark and DreamGen robotics benchmark under VLM-based evaluation.
Pretrained video generators are promising visual world models that exhibit emergent task-solving abilities; however, their reliance on detailed textual descriptions limits their direct use for planning and decision-making. Existing approaches either outsource this reasoning to language or vision-language models, or rely on supervised fine-tuning with paired task-execution videos, which are costly to collect and difficult to scale. We propose a scalable framework that elicits task-solving ability in such models by combining self-distillation with reinforcement learning. Given an unlabeled scene image, a vision-language model generates a candidate task and a detailed step-by-step solution. The solution conditions a pretrained video diffusion model, the Demonstrator; we distill its behavior into an Executor conditioned only on the image and a short task prompt. This transfers execution knowledge from caption-guided generation to instruction-conditioned task solving without curated task-video supervision. We further improve the Executor with reinforcement learning from VLM feedback, exploiting the asymmetry between judging whether a sampled video satisfies a task and generating the solution. Experiments on our proposed WorldTasks-Benchmark and the DreamGen robotics benchmark show that the Executor surpasses the Demonstrator under our VLM-based evaluation protocol and transfers competitively to robotic tasks.