ORM

“Early alignment efforts relied on Outcome Reward Models (ORM)… where feedback is concentrated solely on the final solution; however, this approach faces a severe credit assignment problem”

“Process Reward Models (PRMs) enable fine-grained step-level supervision for model reasoning, addressing traditional Outcome Reward Models (ORMs) limitation of only scoring final outputs.”

“This granularity mismatch leads to inconsistent scoring between partial and complete sequences.”

“Existing works wang2024mathshepherdverifyreinforcellms, lightman2023letsverifystepstep show that PRM outperforms ORM by a considerable margin”

“most of recent work demonstrates that ORMs fall short on complex multi-step reasoning tasks”

“ORM evaluates the whole reasoning process based on the final answer, ignoring intermediate steps”

“Traditional Outcome Reward Models (ORMs)...which assign rewards based solely on the final answer, fail to detect flawed intermediate reasoning.”

“Outcome Reward Models (ORMs) focus solely on evaluating final solutions for correctness, ignoring process optimality and therefore missing costly but non-terminal inefficiencies”

“ORM evaluates the entire reasoning path by assigning a single score to the final solution, whereas PRM provides step-level scores, yielding denser reward signals and generally outperforming ORM.”

“this approach, which provides a sparse reward risks inadvertently validating reasoning trajectories that are flawed, illogical, or factually incorrect, so long as they coincidentally arrive at the correct final output”