MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
This work addresses the challenge of global exploration for embodied agents in vision-and-language navigation, offering a novel approach that enhances zero-shot capabilities.
The paper tackled the problem of vision-and-language navigation by introducing MapGPT, which uses an online linguistic-formed map to provide a global view, achieving state-of-the-art zero-shot performance with ~10% and ~12% improvements in success rates on R2R and REVERIE benchmarks.
Embodied agents equipped with GPT as their brains have exhibited extraordinary decision-making and generalization abilities across various tasks. However, existing zero-shot agents for vision-and-language navigation (VLN) only prompt GPT-4 to select potential locations within localized environments, without constructing an effective "global-view" for the agent to understand the overall environment. In this work, we present a novel map-guided GPT-based agent, dubbed MapGPT, which introduces an online linguistic-formed map to encourage global exploration. Specifically, we build an online map and incorporate it into the prompts that include node information and topological relationships, to help GPT understand the spatial environment. Benefiting from this design, we further propose an adaptive planning mechanism to assist the agent in performing multi-step path planning based on a map, systematically exploring multiple candidate nodes or sub-goals step by step. Extensive experiments demonstrate that our MapGPT is applicable to both GPT-4 and GPT-4V, achieving state-of-the-art zero-shot performance on R2R and REVERIE simultaneously (~10% and ~12% improvements in SR), and showcasing the newly emergent global thinking and path planning abilities of the GPT.