RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents
This addresses the problem of enabling LLM agents to reflect on past experiences for better planning in complex applications like robotics and gaming, representing an incremental advancement in agent capabilities.
The paper tackles the challenge of incorporating past experiences into decision-making for LLM agents by proposing the Retrieval-Augmented Planning (RAP) framework, which dynamically leverages contextual memory to enhance planning, achieving state-of-the-art performance in textual scenarios and improving multimodal agents for embodied tasks.
Owing to recent advancements, Large Language Models (LLMs) can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming, and API integration. However, reflecting past experiences in current decision-making processes, an innate human behavior, continues to pose significant challenges. Addressing this, we propose Retrieval-Augmented Planning (RAP) framework, designed to dynamically leverage past experiences corresponding to the current situation and context, thereby enhancing agents' planning capabilities. RAP distinguishes itself by being versatile: it excels in both text-only and multimodal environments, making it suitable for a wide range of tasks. Empirical evaluations demonstrate RAP's effectiveness, where it achieves SOTA performance in textual scenarios and notably enhances multimodal LLM agents' performance for embodied tasks. These results highlight RAP's potential in advancing the functionality and applicability of LLM agents in complex, real-world applications.