CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval
This work provides a substantial improvement for vision-and-language navigation agents, particularly for tasks requiring long-horizon planning and adaptation to unfamiliar environments, by enabling more effective use of past experiences.
This paper addresses the limitation of LLM-based vision-and-language navigation (VLN) agents in long-horizon and unfamiliar scenarios by introducing a framework that allows them to selectively recall and use prior experiences. The proposed CMMR-VLN framework significantly improves success rates, achieving average gains of 52.9%, 20.9%, and 20.9% over NavGPT, MapGPT, and DiscussNav in simulation, and 200%, 50%, and 50% in real-world tests.
Although large language models (LLMs) are introduced into vision-and-language navigation (VLN) to improve instruction comprehension and generalization, existing LLM- based VLN lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks, limiting their performance in long-horizon and unfamiliar scenarios. In this work, we propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities. Specifically, the CMMR-VLN constructs a multimodal experi- ence memory indexed by panoramic visual images and salient landmarks to retrieve relevant experiences during navigation, introduces a retrieved-augmented generation pipeline to mimick how experienced human navigators leverage priori knowledge, and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases. Comprehensive tests illustrate average success rate improvements of 52.9%, 20.9% and 20.9%, and 200%, 50% and 50% over the NavGPT, the MapGPT, and the DiscussNav in simulation and real tests, respectively eluci- dating the great potential of the CMMR-VLN as a backbone VLN framework.