Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings
This addresses a critical dilemma in sparse-reward settings for reinforcement learning with verifiable rewards, offering a novel solution to improve training efficiency and performance.
The paper tackled the problem of advantage collapse and high-variance gradient estimation in sparse-reward reinforcement learning by introducing Hindsight-Anchored Policy Optimization (HAPO), which achieved asymptotic consistency and enabled the model to surpass static teacher forcing limitations.
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for post-training reasoning models. However, group-based methods such as Group Relative Policy Optimization (GRPO) face a critical dilemma in sparse-reward settings: pure Reinforcement Learning (RL) suffers from advantage collapse and high-variance gradient estimation, while mixed-policy optimization introduces persistent distributional bias. To resolve this dilemma, we introduce Hindsight-Anchored Policy Optimization (HAPO). HAPO employs the Synthetic Success Injection (SSI) operator, a hindsight mechanism that selectively anchors optimization to teacher demonstrations during failure. This injection is governed by a Thompson sampling-inspired gating mechanism, creating an autonomous, self-paced curriculum. Theoretically, we demonstrate that HAPO achieves \textit{asymptotic consistency}: by naturally annealing the teacher signal as the policy improves, HAPO recovers the unbiased on-policy gradient. This ensures off-policy guidance acts as a temporary scaffold rather than a persistent ceiling, enabling the model to surpass the limitations of static teacher forcing.