ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training
This addresses a specific bottleneck in reinforcement learning post-training for LLMs in user-agent interactions like recommendation and brainstorming, offering an incremental improvement over existing methods.
The paper tackled the problem of noisy training signals in multi-candidate LLM training, where existing methods assign the same reward to all candidates, leading to suboptimal exploration; the proposed ShapE-GRPO method improved performance by decomposing set-level rewards into candidate-specific signals, resulting in consistent outperformance over standard GRPO across diverse datasets with accelerated convergence.
In user-agent interaction scenarios such as recommendation, brainstorming, and code suggestion, Large Language Models (LLMs) often generate sets of candidate recommendations where the objective is to maximize the collective utility of the entire set rather than individual candidates independently. However, existing reinforcement learning post-training paradigms, such as Group Relative Policy Optimization (GRPO), typically assign the same set-level scalar reward to every candidate in the set. This leads to noisy training signals where poor candidates free-ride on the high reward produced by a single strong peer, resulting in suboptimal exploration. To address this, we propose Shapley-Enhanced GRPO (ShapE-GRPO). By leveraging the permutation-invariant nature of set-level utility, we derive a Shapley-enhanced formulation from cooperative game theory to decompose set-level rewards into granular, candidate-specific signals. We show that our formulation preserves the fundamental axioms of the Shapley value while remaining computationally efficient with polynomial-time complexity. Empirically, ShapE-GRPO consistently outperforms standard GRPO across diverse datasets with accelerated convergence during training.