One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry
This addresses a fundamental limitation in group-based RL for researchers and practitioners by providing a more adaptive approach to trajectory aggregation.
The paper tackles the problem of fixed aggregation geometry in group-based reinforcement learning by introducing Power-Mean Policy Optimization (PMPO), a framework that dynamically adjusts aggregation via a power-mean exponent p, outperforming baselines on multiple mathematical reasoning benchmarks.
Group-based reinforcement learning has evolved from the arithmetic mean of GRPO to the geometric mean of GMPO. While GMPO improves stability by constraining a conservative objective, it shares a fundamental limitation with GRPO: reliance on a fixed aggregation geometry that ignores the evolving and heterogeneous nature of each trajectory. In this work, we unify these approaches under Power-Mean Policy Optimization (PMPO), a generalized framework that parameterizes the aggregation geometry via the power-mean geometry exponent p. Within this framework, GRPO and GMPO are recovered as special cases. Theoretically, we demonstrate that adjusting p modulates the concentration of gradient updates, effectively reweighting tokens based on their advantage contribution. To determine p adaptively, we introduce a Clip-aware Effective Sample Size (ESS) mechanism. Specifically, we propose a deterministic rule that maps a trajectory clipping fraction to a target ESS. Then, we solve for the specific p to align the trajectory induced ESS with this target one. This allows PMPO to dynamically transition between the aggressive arithmetic mean for reliable trajectories and the conservative geometric mean for unstable ones. Experiments on multiple mathematical reasoning benchmarks demonstrate that PMPO outperforms strong baselines.