Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search
This work addresses the problem of creating reliable and interpretable world models for reinforcement learning practitioners, though it is incremental as it builds on existing LLM and code generation methods.
The authors tackled the challenge of generating precise and efficient world models as Python code for model-based reinforcement learning by proposing GIF-MCTS, a code generation strategy using LLMs guided by Monte Carlo Tree Search. They introduced the Code World Models Benchmark and demonstrated that GIF-MCTS outperformed baselines, enabling model-based RL agents with improved sample efficiency and inference speed.
In this work we consider Code World Models, world models generated by a Large Language Model (LLM) in the form of Python code for model-based Reinforcement Learning (RL). Calling code instead of LLMs for planning has potential to be more precise, reliable, interpretable, and extremely efficient. However, writing appropriate Code World Models requires the ability to understand complex instructions, to generate exact code with non-trivial logic and to self-debug a long program with feedback from unit tests and environment trajectories. To address these challenges, we propose Generate, Improve and Fix with Monte Carlo Tree Search (GIF-MCTS), a new code generation strategy for LLMs. To test our approach in an offline RL setting, we introduce the Code World Models Benchmark (CWMB), a suite of program synthesis and planning tasks comprised of 18 diverse RL environments paired with corresponding textual descriptions and curated trajectories. GIF-MCTS surpasses all baselines on the CWMB and two other benchmarks, and we show that the Code World Models synthesized with it can be successfully used for planning, resulting in model-based RL agents with greatly improved sample efficiency and inference speed.