DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
Improves credit assignment and training stability for RL-based fine-tuning of LLMs on complex reasoning tasks, addressing a key bottleneck in current methods.
DGPO introduces a critic-free RL framework for LLM reasoning that uses distribution deviation as a guiding signal instead of a rigid KL penalty, enabling fine-grained credit assignment at the token level. It achieves 8.3% higher accuracy on MATH-500 and 6.7% on GSM8K compared to GRPO, with 40% faster convergence.
Reinforcement learning is crucial for aligning large language models to perform complex reasoning tasks. However, current algorithms such as Group Relative Policy Optimization suffer from coarse grained, sequence level credit assignment, which severely struggles to isolate pivotal reasoning steps within long Chain of Thought generations. Furthermore, the standard unbounded Kullback Leibler divergence penalty induces severe gradient instability and mode seeking conservatism, ultimately stifling the discovery of novel reasoning trajectories. To overcome these limitations, we introduce Distribution Guided Policy Optimization, a novel critic free reinforcement learning framework that reinterprets distribution deviation as a guiding signal rather than a rigid penalty.