Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
For practitioners training LLMs on math reasoning, this provides a zero-overhead method to improve RLVR by leveraging geometric structure in hidden states.
RLVR training for math reasoning wastes information by reducing correct rollouts to a single reward bit. Hidden-Align, an auxiliary loss aligning hidden states at the anchor token, boosts pass@1 by 3.8–6.2 pp over DAPO on eight benchmarks across three model scales.
Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.