Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
This provides a principled mechanism for dynamically controlling exploration-exploitation trade-offs in RL-tuned LLMs, addressing a key bottleneck in reasoning-intensive applications.
The paper tackles the problem of explicitly steering reinforcement learning training toward exploration or exploitation in large language models by introducing Token Hidden Reward (THR), a token-level metric that quantifies each token's influence on correct responses. The result is a THR-guided reweighting algorithm that improves greedy-decoding accuracy by 1-3% for exploitation and boosts Pass@K accuracy by 2-5% for exploration on math reasoning benchmarks.
Reinforcement learning with verifiable rewards has significantly advanced the reasoning capabilities of large language models, yet how to explicitly steer training toward exploration or exploitation remains an open problem. We introduce Token Hidden Reward (THR), a token-level metric that quantifies each token's influence on the likelihood of correct responses under Group Relative Policy Optimization (GRPO). We find that training dynamics are dominated by a small subset of tokens with high absolute THR values. Most interestingly, tokens with positive THR strengthen confidence in correct outputs, thus favoring exploitation, while tokens with negative THR preserve probability mass for alternative outputs, enabling exploration. This insight suggests a natural intervention: a THR-guided reweighting algorithm that modulates GRPO's learning signals to explicitly bias training toward exploitation or exploration. We validate the efficacy of this algorithm on diverse math reasoning benchmarks. By amplifying tokens with positive THR value and weakening negative ones, our algorithm improves greedy-decoding accuracy, favoring exploitation. The reverse strategy yields consistent gains in Pass@K accuracy, favoring exploration. We further demonstrate that our algorithm integrates seamlessly with other RL objectives such as GSPO and generalizes across architectures including Llama. These findings establish THR as a principled and fine-grained mechanism for dynamically controlling exploration and exploitation in RL-tuned LLMs, providing new tools for targeted fine-tuning in reasoning-intensive applications.