Adversarial Training for Process Reward Models
This addresses the challenge of enhancing reasoning in LLMs for tasks like mathematical reasoning by reducing annotation costs and improving generalization, though it is incremental as it builds on existing PRM methods.
The paper tackles the problem of expensive manual step-level annotation and poor generalization in Process Reward Models (PRMs) for LLMs by introducing Adversarially Trained PRMs (APRM), which uses adversarial training to generate harder negatives, improving solver accuracy by +3.4 percentage points on average and +5.3 percentage points on out-of-distribution tasks.
Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs (\texttt{APRM}), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$), while $R$ concurrently learns to detect them. This interaction yields progressively harder negatives for $R$, improving its robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, \texttt{APRM} improves solver accuracy by $+3.4$ percentage points (pp) over the strongest PRM baseline. \texttt{APRM} achieves gains of $+5.3$ pp on out-of-distribution tasks.