Memory Transfer Planning: LLM-driven Context-Aware Code Adaptation for Robot Manipulation
This addresses the challenge of adapting LLM-based systems to new robotic environments, offering a practical solution for enhancing transferability, though it is incremental by building on existing retrieval and planning methods.
The paper tackled the problem of limited transferability in LLM-driven robot manipulation by introducing Memory Transfer Planning (MTP), which uses successful code examples as in-context guidance to adapt plans without retraining, resulting in improved success rates and adaptability across simulation and physical robot experiments.
Large language models (LLMs) are increasingly explored in robot manipulation, but many existing methods struggle to adapt to new environments. Many systems require either environment-specific policy training or depend on fixed prompts and single-shot code generation, leading to limited transferability and manual re-tuning. We introduce Memory Transfer Planning (MTP), a framework that leverages successful control-code examples from different environments as procedural knowledge, using them as in-context guidance for LLM-driven planning. Specifically, MTP (i) generates an initial plan and code using LLMs, (ii) retrieves relevant successful examples from a code memory, and (iii) contextually adapts the retrieved code to the target setting for re-planning without updating model parameters. We evaluate MTP on RLBench, CALVIN, and a physical robot, demonstrating effectiveness beyond simulation. Across these settings, MTP consistently improved success rate and adaptability compared with fixed-prompt code generation, naive retrieval, and memory-free re-planning. Furthermore, in hardware experiments, leveraging a memory constructed in simulation proved effective. MTP provides a practical approach that exploits procedural knowledge to realize robust LLM-based planning across diverse robotic manipulation scenarios, enhancing adaptability to novel environments and bridging simulation and real-world deployment.