SignScene: Visual Sign Grounding for Mapless Navigation
This work is significant for robotics, enabling mapless navigation in unfamiliar environments by interpreting visual signs, which is an incremental step towards more autonomous robots.
This paper addresses the problem of robots using visual signs for mapless navigation by formalizing "sign grounding," which maps semantic instructions on signs to scene elements and navigational actions. The proposed SignScene representation, which presents sign and scene information to Vision-Language Models, achieved 88% grounding accuracy on a dataset of 114 queries and enabled real-world mapless navigation.
Navigational signs enable humans to navigate unfamiliar environments without maps. This work studies how robots can similarly exploit signs for mapless navigation in the open world. A central challenge lies in interpreting signs: real-world signs are diverse and complex, and their abstract semantic contents need to be grounded in the local 3D scene. We formalize this as sign grounding, the problem of mapping semantic instructions on signs to corresponding scene elements and navigational actions. Recent Vision-Language Models (VLMs) offer the semantic common-sense and reasoning capabilities required for this task, but are sensitive to how spatial information is represented. We propose SignScene, a sign-centric spatial-semantic representation that captures navigation-relevant scene elements and sign information, and presents them to VLMs in a form conducive to effective reasoning. We evaluate our grounding approach on a dataset of 114 queries collected across nine diverse environment types, achieving 88% grounding accuracy and significantly outperforming baselines. Finally, we demonstrate that it enables real-world mapless navigation on a Spot robot using only signs.