Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning
This addresses the issue of opaque and biased reasoning in medical evidence synthesis for researchers and practitioners, though it is incremental as it builds on existing reinforcement learning with verifiable rewards.
The paper tackled the problem of improving large language models' structured reasoning by introducing Verifiable Process Reward Models (VPRMs), which use deterministic, rule-based verifiers to check intermediate steps, resulting in up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards.
Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification signals, such as unit tests for code or exact-match checks for mathematics. In parallel, process supervision has long been explored as a way to shape the intermediate reasoning behaviour of LLMs, but existing approaches rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking. To address this gap, we introduce Verifiable Process Reward Models (VPRMs), a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers. We apply VPRMs to risk-of-bias assessment for medical evidence synthesis, a domain where guideline-defined criteria and rule-based decision paths enable programmatic verification of reasoning traces. Across multiple datasets, we find that VPRMs generate reasoning that adheres closely to domain rules and achieve substantially higher coherence between step-level decisions and final labels. Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence.