Rethinking Reward Miscalibration of GRPO in Agentic RL
This addresses training instability in autonomous agents for long-horizon tasks, though it appears incremental as it builds on existing GRPO methods.
The paper tackles reward miscalibration in agentic reinforcement learning by showing that outcome-based rewards should punish flawed actions, not reinforce them, and identifies gradient coupling as a key issue where similar samples interfere with training. It proposes training the actor to classify good/bad actions to separate embeddings, with experiments demonstrating effectiveness.
Building autonomous agents capable of solving long-horizon, real-world tasks has garnered significant research interest. But outcome based rewards may cause reward miscalibration which means it might mistakenly allocate positive reward to flawed middle steps which is regarded as the key reason making the bad actions being reinforced during training. However we reveal that outcome based reward ensures expected negative advantage for those flawed middle steps, which means the flawed actions should be punished during training. Even accounting for the ``squeezing effect", the probability mass of good actions should increase and the actor should gradually get rid of harmful actions. This shows that flawed actions should be punished during training. We further identify gradient coupling between similar samples as a key issue in agentic RL, the input prompt is extremely similar and the output action space is limited, therefore during training, gradients from well-performing samples can inadvertently strengthen suboptimal or incorrect actions due to similar input observation and output actions. We show that with gradient coupling, some flawed actions might be enhanced. To address this, we propose training the actor to classify good or bad actions to separate the embedding of good/bad actions and alleviate the gradient interference, extensive experiments shows its effectiveness.