GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards
For practitioners fine-tuning LLMs on reasoning tasks, GRAIL offers a simple, process-supervision-free way to improve reasoning alignment.
GRAIL introduces a token-wise advantage reweighting method using gradient-activation saliency to improve reinforcement learning with verifiable rewards for LLMs, achieving average accuracy improvements of 3.60% and Pass@3 improvements of 3.05% over GRPO across multiple model families.
Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical reasoning in Large Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained reasoning alignment can be achieved without process-level supervision.