From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models
This work addresses the challenge of enabling robots to generalize to novel tasks with minimal demonstrations, though it builds incrementally on existing vision-language models and planning methods.
The paper tackles the problem of learning symbolic world models for long-horizon decision-making in robotics by leveraging pretrained vision-language models to propose and evaluate visual predicates, enabling zero-shot generalization to novel goals and settings. The result is demonstrated empirically in simulation and real-world experiments, showing aggressive generalization across diverse object types, arrangements, numbers, backgrounds, and longer horizons than seen during training.
Our aim is to learn to solve long-horizon decision-making problems in complex robotics domains given low-level skills and a handful of short-horizon demonstrations containing sequences of images. To this end, we focus on learning abstract symbolic world models that facilitate zero-shot generalization to novel goals via planning. A critical component of such models is the set of symbolic predicates that define properties of and relationships between objects. In this work, we leverage pretrained vision language models (VLMs) to propose a large set of visual predicates potentially relevant for decision-making, and to evaluate those predicates directly from camera images. At training time, we pass the proposed predicates and demonstrations into an optimization-based model-learning algorithm to obtain an abstract symbolic world model that is defined in terms of a compact subset of the proposed predicates. At test time, given a novel goal in a novel setting, we use the VLM to construct a symbolic description of the current world state, and then use a search-based planning algorithm to find a sequence of low-level skills that achieves the goal. We demonstrate empirically across experiments in both simulation and the real world that our method can generalize aggressively, applying its learned world model to solve problems with a wide variety of object types, arrangements, numbers of objects, and visual backgrounds, as well as novel goals and much longer horizons than those seen at training time.