Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval
This addresses the challenge of costly training and data collection for robot planning, offering a flexible and efficient solution for autonomous navigation applications, though it is incremental as it builds on existing VLM capabilities.
This study tackles the problem of high-level robot planning for autonomous navigation by introducing Select2Plan, a training-free framework that uses Vision-Language Models with Visual Question-Answering and In-Context Learning, eliminating the need for fine-tuning and reducing data requirements. The result shows a 50% improvement in Third-Person View navigation compared to a baseline VLM and performance comparable to trained models in First-Person View with only 20 demonstrations.
This study explores the potential of off-the-shelf Vision-Language Models (VLMs) for high-level robot planning in the context of autonomous navigation. Indeed, while most of existing learning-based approaches for path planning require extensive task-specific training/fine-tuning, we demonstrate how such training can be avoided for most practical cases. To do this, we introduce Select2Plan (S2P), a novel training-free framework for high-level robot planning which completely eliminates the need for fine-tuning or specialised training. By leveraging structured Visual Question-Answering (VQA) and In-Context Learning (ICL), our approach drastically reduces the need for data collection, requiring a fraction of the task-specific data typically used by trained models, or even relying only on online data. Our method facilitates the effective use of a generally trained VLM in a flexible and cost-efficient way, and does not require additional sensing except for a simple monocular camera. We demonstrate its adaptability across various scene types, context sources, and sensing setups. We evaluate our approach in two distinct scenarios: traditional First-Person View (FPV) and infrastructure-driven Third-Person View (TPV) navigation, demonstrating the flexibility and simplicity of our method. Our technique significantly enhances the navigational capabilities of a baseline VLM of approximately 50% in TPV scenario, and is comparable to trained models in the FPV one, with as few as 20 demonstrations.