Foundation Models as World Models: A Foundational Study in Text-Based GridWorlds
This work addresses the challenge of sample-efficient reinforcement learning for applications with expensive interactions, though it is incremental as it builds on existing foundation model capabilities.
The study tackled the problem of improving sample efficiency in reinforcement learning for real-world applications by evaluating two strategies using foundation models: foundation world models (FWMs) for simulation and foundation agents (FAs) for decision-making in text-based grid-worlds. The results showed that better large language models (LLMs) lead to improved FWMs and FAs, with FAs providing excellent policies in simple environments and FWMs coupled with reinforcement learning agents being promising for complex settings.
While reinforcement learning from scratch has shown impressive results in solving sequential decision-making tasks with efficient simulators, real-world applications with expensive interactions require more sample-efficient agents. Foundation models (FMs) are natural candidates to improve sample efficiency as they possess broad knowledge and reasoning capabilities, but it is yet unclear how to effectively integrate them into the reinforcement learning framework. In this paper, we anticipate and, most importantly, evaluate two promising strategies. First, we consider the use of foundation world models (FWMs) that exploit the prior knowledge of FMs to enable training and evaluating agents with simulated interactions. Second, we consider the use of foundation agents (FAs) that exploit the reasoning capabilities of FMs for decision-making. We evaluate both approaches empirically in a family of grid-world environments that are suitable for the current generation of large language models (LLMs). Our results suggest that improvements in LLMs already translate into better FWMs and FAs; that FAs based on current LLMs can already provide excellent policies for sufficiently simple environments; and that the coupling of FWMs and reinforcement learning agents is highly promising for more complex settings with partial observability and stochastic elements.