Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
For multi-agent game AI, this work demonstrates that LLM-based planning can replace manual rule engineering while achieving competitive performance and higher perceived believability.
The paper proposes a hierarchical architecture combining a pretrained LLM for strategic planning with RL skill policies for execution in multi-agent games. In a 2v2 King of the Hill environment, the LLM+RL system achieves a 46.4% win rate, statistically equivalent to hand-crafted behavior trees (51.5%), while outperforming end-to-end RL, and 60% of participants rated its behavior as most human-like.
Reinforcement learning (RL) has achieved strong performance in sequential decision-making, yet scaling to complex multi-agent environments remains challenging due to sparse rewards, large state-action spaces, and the difficulty of learning coordinated strategies. We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a centralized strategic controller that selects among specialized RL skill policies for a team of agents, while RL policies handle reactive low-level execution. We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and \emph{``Flat''} RL (end-to-end training without skill decomposition) baselines. The LLM+RL system achieves task performance statistically equivalent to hand-crafted BT (46.4\% vs 51.5\% win rate, $p=0.103$) while both significantly outperform Flat RL trained without skill decomposition. A user study ($n=15$) reveals that 60\% of participants perceive LLM+RL agents as the most human-like ($p=0.027$), citing behavioral adaptability and tactical variability. These results demonstrate that pretrained LLM reasoning can effectively orchestrate pretrained RL skills, achieving competitive multi-agent coordination and superior perceived believability without manual rule engineering.