Learning Universal Policies via Text-Guided Video Generation
This work proposes a novel approach to building more versatile AI agents by leveraging text-guided video generation, potentially advancing robotics and AI planning, though it appears incremental in applying image synthesis tools to decision-making.
The paper tackles the problem of constructing general-purpose agents for sequential decision-making by casting it as a text-conditioned video generation task, where a planner synthesizes future frames to represent planned actions and extracts control actions, enabling combinatorial generalization to novel goals and learning across diverse robot manipulation tasks.
A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents. Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able to naturally and combinatorially generalize to novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available videos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots.