Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
For LLM reasoning tasks with sparse rewards, HSD provides a dense per-token signal that outperforms existing methods, particularly on terse-answer tasks.
Hindsight Self-Distillation (HSD) improves token-level credit assignment in LLM reasoning by conditioning the teacher on a successful peer rollout, achieving state-of-the-art results on math and code benchmarks (e.g., AIME) with gains over GRPO variants and on-policy distillation baselines.
Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal. But the common choice of a ground-truth answer is only an endpoint cue: on terse-answer tasks, the teacher falls silent at the intermediate positions where path-level guidance matters most. We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout drawn from the current training group. Such a peer is an exact sample from the success-conditioned policy, requiring no additional sampled rollouts. By providing a full successful continuation rather than only the final answer, the resulting credit signal concentrates at the divergence position between a failed rollout and a successful peer. Across Qwen3-8B and Qwen3-32B on math and code benchmarks, HSD obtains the best result against GRPO variants and on-policy distillation baselines, with the largest gains on terse-answer tasks such as AIME.