Arbitrary Entropy Policy Optimization: Entropy Is Controllable in Reinforcement Fine-tuning
This work addresses a critical bottleneck in improving reasoning capabilities of LLMs, offering a novel solution for controllable exploration in reinforcement fine-tuning.
The paper tackles the problem of entropy collapse in reinforcement fine-tuning for large language models, where existing methods like GRPO suffer from vanishing exploration. The proposed AEPO method stabilizes entropy at arbitrary target levels, reveals a non-monotonic relationship between entropy and performance, and generalizes to broader regularization paradigms.
Reinforcement fine-tuning (RFT) is essential for enhancing the reasoning capabilities of large language models (LLM), yet the widely adopted Group Relative Policy Optimization (GRPO) suffers from entropy collapse, where entropy monotonically decreases, exploration vanishes, and policies converge prematurely. Existing entropy-regularized methods only partially alleviate this issue while introducing bias and instability, leaving entropy control unresolved and the connection between entropy, exploration, and performance unclear. We propose Arbitrary Entropy Policy Optimization (AEPO), which eliminates entropy collapse by replacing entropy bonuses with REINFORCE policy gradient on temperature-adjusted distributions and stabilizing entropy through temperature regulation. AEPO integrates three key designs: policy gradient as regularization, distribution as regularization, and REINFORCE as regularization, enabling precise entropy control without distorting optimization. Experiments demonstrate three major contributions: AEPO (1) stabilizes entropy at arbitrary target levels, effectively removing collapse in GRPO; (2) reveals a non-monotonic relation where performance first improves then declines with increasing entropy, clarifying the link between entropy, exploration, and reasoning; and (3) generalizes beyond entropy, providing a broader RFT paradigm where superior target distributions can serve as REINFORCE regularizers.