Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification
This addresses a systemic pathology in reinforcement learning for verifiable rewards, offering a novel solution to prevent exploration collapse, though it appears incremental as it builds on existing regularization methods.
The paper tackles the problem of Recursive Space Contraction in reinforcement learning, where exploration collapses due to conflicting dynamics, and proposes Anchored Policy Optimization to shift from shape matching to support coverage, resulting in improved accuracy and diversity on benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) is increasingly viewed as a tree pruning mechanism. However, we identify a systemic pathology termed Recursive Space Contraction (RSC), an irreversible collapse driven by the combined dynamics of positive sharpening and negative squeezing, where the sampling probability of valid alternatives vanishes. While Kullback-Leibler (KL) regularization aims to mitigate this, it imposes a rigid Shape Matching constraint that forces the policy to mimic the reference model's full density, creating a gradient conflict with the sharpening required for correctness. We propose Anchored Policy Optimization (APO), shifting the paradigm from global Shape Matching to Support Coverage. By defining a Safe Manifold based on the reference model's high-confidence support, APO permits aggressive sharpening for efficiency while selectively invoking a restorative force during error correction to prevent collapse. We theoretically derive that APO serves as a gradient-aligned mechanism to maximize support coverage, enabling an Elastic Recovery that re-inflates valid branches. Empirical evaluations on mathematical benchmarks demonstrate that APO breaks the accuracy-diversity trade-off, significantly improving Pass@1 while restoring the Pass@K diversity typically lost by standard policy gradient methods.