From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
This addresses the challenge of catastrophic forgetting and lack of adaptability in LLM agents for autonomous task-solving in complex environments, representing a novel method rather than an incremental improvement.
The paper tackles the problem of LLM agents' limited ability to utilize prior experiences for decision-making by introducing a trainable graph memory framework that abstracts trajectories into strategic meta-cognition, resulting in improved strategic reasoning performance and robust generalization, with consistent benefits during RL training.
Large Language Models (LLMs) based agents have demonstrated remarkable potential in autonomous task-solving across complex, open-ended environments. A promising approach for improving the reasoning capabilities of LLM agents is to better utilize prior experiences in guiding current decisions. However, LLMs acquire experience either through implicit memory via training, which suffers from catastrophic forgetting and limited interpretability, or explicit memory via prompting, which lacks adaptability. In this paper, we introduce a novel agent-centric, trainable, multi-layered graph memory framework and evaluate how context memory enhances the ability of LLMs to utilize parametric information. The graph abstracts raw agent trajectories into structured decision paths in a state machine and further distills them into high-level, human-interpretable strategic meta-cognition. In order to make memory adaptable, we propose a reinforcement-based weight optimization procedure that estimates the empirical utility of each meta-cognition based on reward feedback from downstream tasks. These optimized strategies are then dynamically integrated into the LLM agent's training loop through meta-cognitive prompting. Empirically, the learnable graph memory delivers robust generalization, improves LLM agents' strategic reasoning performance, and provides consistent benefits during Reinforcement Learning (RL) training.