Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Improves long-horizon task performance for LLM-based agents by enabling self-generated progress signals.
LLM-based agents lack metacognitive awareness of task progress, hindering long-horizon scaling. RePro, a retrospective progress-aware training framework, achieves up to 12% absolute success rate gains on WebShop, ALFWorld, and Sokoban.
LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders long-horizon scaling. A pilot study reveals that online progress prompting hurts performance while retrospective demonstrations help, yet this capability cannot emerge from outcome-reward training alone. We present RePro, Retrospective Progress-Aware Training, a framework that trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm: the agent executes actions online, then retrospectively reassesses its step-wise progress given the completed trajectory and known outcome. RePro initializes with a Retrospection Warmup that teaches reflection format from minimal external demonstrations, then further trains through RePro-PO with a composite reward that produces self-generated signals without continuous external supervision. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to $12\%$ absolute success rate gains.