Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
This work addresses a key bottleneck in enhancing reasoning abilities for LLMs, offering a more stable and effective method for entropy regularization in RLVR, though it is incremental as it builds on existing regularization techniques.
The paper tackles policy entropy collapse in Reinforcement Learning with Verifiable Rewards for Large Language Models by proposing Adaptive Entropy Regularization, which dynamically adjusts the entropy coefficient based on task difficulty and maintains moderate entropy levels, resulting in improved reasoning accuracy and exploration on mathematical reasoning benchmarks.
Reasoning ability has become a defining capability of Large Language Models (LLMs), with Reinforcement Learning with Verifiable Rewards (RLVR) emerging as a key paradigm to enhance it. However, RLVR training often suffers from policy entropy collapse, where the policy becomes overly deterministic, hindering exploration and limiting reasoning performance. While entropy regularization is a common remedy, its effectiveness is highly sensitive to the fixed coefficient, making it unstable across tasks and models. In this work, we revisit entropy regularization in RLVR and argue that its potential has been largely underestimated. Our analysis shows that (i) tasks of varying difficulty demand distinct exploration intensities, and (ii) balanced exploration may require the policy entropy to be maintained within a moderate range below its initial level. Therefore, we propose Adaptive Entropy Regularization (AER)--a framework that dynamically balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment. Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.