Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
This work provides a method to improve the training stability and generalization of LLMs for researchers and developers working on multi-turn agentic systems, addressing a known limitation of self-distillation techniques.
This paper addresses the issue of exposure bias in On-Policy Self-Distillation (OPSD) for post-training Large Language Models (LLMs) in multi-turn agentic settings, which leads to reasoning route convergence. They introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates OPSD using contrastive learning to distinguish between positive and negative positions, thereby preserving fine-grained optimization signals. CRPO consistently outperforms existing baselines across 13 reasoning and deep-search benchmarks, enhancing training stability and generalization.
Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.