FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
This work addresses the challenge of reward-free embodied decision-making for AI agents, offering a novel integration approach that improves task generalization, though it is incremental in combining existing model types.
The authors tackled the problem of enabling open-ended task solving in embodied environments by integrating Foundation Models (FMs) and World Models (WMs) to ground FM representations in WM state space, resulting in superior performance on multi-task offline visual control benchmarks, particularly in handling complex observations or domain gaps.
Foundation Models (FMs) and World Models (WMs) offer complementary strengths in task generalization at different levels. In this work, we propose FOUNDER, a framework that integrates the generalizable knowledge embedded in FMs with the dynamic modeling capabilities of WMs to enable open-ended task solving in embodied environments in a reward-free manner. We learn a mapping function that grounds FM representations in the WM state space, effectively inferring the agent's physical states in the world simulator from external observations. This mapping enables the learning of a goal-conditioned policy through imagination during behavior learning, with the mapped task serving as the goal state. Our method leverages the predicted temporal distance to the goal state as an informative reward signal. FOUNDER demonstrates superior performance on various multi-task offline visual control benchmarks, excelling in capturing the deep-level semantics of tasks specified by text or videos, particularly in scenarios involving complex observations or domain gaps where prior methods struggle. The consistency of our learned reward function with the ground-truth reward is also empirically validated. Our project website is https://sites.google.com/view/founder-rl.