Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learning
This addresses a critical gap for researchers and practitioners in AI and robotics by enhancing the reliability of LLM-based planning in complex environments, though it is incremental as it builds on existing LLM and RL methods.
The paper tackled the problem of poor planning-execution alignment in LLM-guided reinforcement learning by proposing a framework that integrates environment-specific subgoal graphs and multi-LLM planning, resulting in improved performance on 22 diverse tasks in an open-world game.
Large language models (LLMs) offer strong high-level planning capabilities for reinforcement learning (RL) by decomposing tasks into subgoals. However, their practical utility is limited by poor planning-execution alignment, which reflects a critical gap between abstract plans and actionable, environment-compatible behaviors. This misalignment arises from two interrelated limitations: (1) LLMs often produce subgoals that are semantically plausible but infeasible or irrelevant in the target environment due to insufficient grounding in environment-specific knowledge, and (2) single-LLM planning conflates generation with self-verification, resulting in overconfident yet unreliable subgoals that frequently fail during execution. To address these challenges, we propose Subgoal Graph-Augmented Actor-Critic-Refiner (SGA-ACR), a framework that integrates an environment-specific subgoal graph and structured entity knowledge with a multi-LLM planning pipeline that explicitly separates generation, critique, and refinement to produce executable and verifiable subgoals. A subgoal tracker further monitors execution progress, provides auxiliary rewards, and adaptively updates the subgoal graph to maintain alignment between plans and actions. Experimental results on 22 diverse tasks in the open-world game "Crafter" demonstrate the effectiveness of our proposed method.