Group Policy Gradient
This work addresses efficiency issues in reinforcement learning for researchers and practitioners by offering a more resource-friendly alternative to existing methods like PPO, though it is incremental as it builds on prior approaches like GRPO.
The paper tackles the problem of reducing computational costs in policy-gradient methods for reinforcement learning by introducing Group Policy Gradient (GPG), a critic-free estimator that replaces learned value functions with group-based Monte Carlo advantage estimation, and demonstrates that GPG matches or outperforms PPO on standard benchmarks while improving computational efficiency.
We introduce Group Policy Gradient (GPG), a family of critic-free policy-gradient estimators for general MDPs. Inspired by the success of GRPO's approach in Reinforcement Learning from Human Feedback (RLHF), GPG replaces a learned value function with a group-based Monte Carlo advantage estimator, removing the memory, compute, and hyperparameter costs of training a critic while preserving PPO's clipped-objective structure. We prove the consistency of the GPG estimator, analyze the bias-variance tradeoffs, and demonstrate empirically that GPG matches or outperforms PPO on standard benchmarks. GPG makes better use of parallel simulations, which, together with its critic-free design, results in more efficient use of computational resources than PPO.