TRE: Encouraging Exploration in the Trust Region
This addresses a specific bottleneck in RL for LLMs, offering a targeted improvement for tasks requiring coherent reasoning.
The paper tackles the problem of entropy regularization degrading performance in Large Language Models (LLMs) due to cumulative tail risk, and proposes Trust Region Entropy (TRE) to encourage exploration within the model's trust region, resulting in consistent outperformance over baselines across tasks like MATH, Countdown, and HH.
Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region.