Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas
This work highlights a critical bias in outdoor VLN systems, showing they rely heavily on graph-specific features rather than visual understanding, which is an incremental but important finding for developing more robust navigation agents.
The paper investigated generalization in vision and language navigation (VLN) for outdoor scenarios, finding that performance gains on unseen outdoor areas primarily come from graph-specific features like junction type embeddings, while image information plays a minor role, revealing a bias toward urban environment graph representations.
Vision and language navigation (VLN) is a challenging visually-grounded language understanding task. Given a natural language navigation instruction, a visual agent interacts with a graph-based environment equipped with panorama images and tries to follow the described route. Most prior work has been conducted in indoor scenarios where best results were obtained for navigation on routes that are similar to the training routes, with sharp drops in performance when testing on unseen environments. We focus on VLN in outdoor scenarios and find that in contrast to indoor VLN, most of the gain in outdoor VLN on unseen data is due to features like junction type embedding or heading delta that are specific to the respective environment graph, while image information plays a very minor role in generalizing VLN to unseen outdoor areas. These findings show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments.