CoEx -- Co-evolving World-model and Exploration
This addresses the issue of dynamic world model updates in LLM agents for improved planning and exploration, representing a novel method for a known bottleneck.
The paper tackles the problem of LLM agents relying on static world models that become misaligned with the true state, leading to erroneous plans, by introducing CoEx, a hierarchical agent architecture that co-evolves planning with a dynamically updated world model, and it outperforms existing paradigms in planning and exploration across diverse scenarios like ALFWorld, PDDL, and Jericho.
Planning in modern LLM agents relies on the utilization of LLM as an internal world model, acquired during pretraining. However, existing agent designs fail to effectively assimilate new observations into dynamic updates of the world model. This reliance on the LLM's static internal world model is progressively prone to misalignment with the underlying true state of the world, leading to the generation of divergent and erroneous plans. We introduce a hierarchical agent architecture, CoEx, in which hierarchical state abstraction allows LLM planning to co-evolve with a dynamically updated model of the world. CoEx plans and interacts with the world by using LLM reasoning to orchestrate dynamic plans consisting of subgoals, and its learning mechanism continuously incorporates these subgoal experiences into a persistent world model in the form of a neurosymbolic belief state, comprising textual inferences and code-based symbolic memory. We evaluate our agent across a diverse set of agent scenarios involving rich environments and complex tasks including ALFWorld, PDDL, and Jericho. Our experiments show that CoEx outperforms existing agent paradigms in planning and exploration.