Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
This work improves policy optimization for large language models on long-horizon agentic tasks by addressing a key issue in advantage estimation, offering a strong specific gain for researchers and practitioners in agentic RL.
This paper addresses the issue of context inconsistency in stepwise group-based policy optimization for long-horizon agentic tasks, which leads to biased advantage estimation. The proposed Hierarchy-of-Groups Policy Optimization (HGPO) method assigns each step to multiple hierarchical groups based on historical context consistency, computes distinct advantages within each group, and aggregates them adaptively. HGPO significantly outperforms existing agentic RL methods on ALFWorld and WebShop tasks using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct.
Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large language models on long-horizon agentic tasks. To enable more fine-grained policy updates, recent research has increasingly shifted toward stepwise group-based policy optimization, which treats each step in a rollout trajectory independently while using a memory module to retain historical context. However, we find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead to severely biased advantage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose Hierarchy-of-Groups Policy Optimization (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step, HGPO computes distinct advantages within each group and aggregates them with an adaptive weighting scheme. In this way, HGPO can achieve a favorable bias-variance trade-off in stepwise advantage estimation, without extra models or rollouts. Evaluations on two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, show that HGPO significantly outperforms existing agentic RL methods under the same computational constraints. Code is available at https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo.