ProcMEM: Learning Reusable Procedural Memory from Experience via Non-Parametric PPO for LLM Agents
This addresses computational redundancy and execution instability for LLM agents in sequential decision-making, representing a novel method for a known bottleneck.
The paper tackles the problem of LLM-driven agents inefficiently re-deriving solutions in recurring scenarios by proposing ProcMEM, a framework that learns reusable procedural memory from experience without parameter updates, achieving superior reuse rates and significant performance gains with extreme memory compression.
LLM-driven agents demonstrate strong performance in sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and execution instability. To bridge this gap, we propose ProcMEM, a framework that enables agents to autonomously learn procedural memory from interaction experiences without parameter updates. By formalizing a Skill-MDP, ProcMEM transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, ProcMEM sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that ProcMEM achieves superior reuse rates and significant performance gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how ProcMEM transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.