Process Reward Models for LLM Agents: Practical Framework and Directions
This work addresses the problem of efficient and scalable training of LLM agents for researchers and developers in the field of natural language processing and reinforcement learning.
The authors tackled the problem of training LLM agents to continually improve through interactions, achieving a result where small 3B models outperform strong GPT-4o baselines on the ALFWorld benchmark. The AgentPRM and InversePRM frameworks demonstrate improved performance with minimal modifications to existing RLHF pipelines.
We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.