Large Language Models Are Semi-Parametric Reinforcement Learning Agents
This addresses the challenge of improving LLM-based reinforcement learning agents' adaptability and learning efficiency without fine-tuning, though it appears incremental in combining existing concepts of memory and RL.
The paper tackles the problem of enabling LLM-based agents to learn from past experiences across different tasks by proposing REMEMBERER, a framework that equips LLMs with a long-term experience memory updated via reinforcement learning, achieving 4% and 2% higher success rates than prior state-of-the-art methods on two task sets.
Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as REMEMBERER. By equipping the LLM with a long-term experience memory, REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed REMEMBERER constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of REMEMBERER.