Uncertainty-Aware Gaussian Map for Vision-Language Navigation
This work addresses the problem of perceptual uncertainty in VLN agents, which is a known bottleneck for reliable navigation in 3D environments, by providing a unified uncertainty-aware representation.
The paper introduces a method for Vision-Language Navigation that explicitly models geometric, semantic, and appearance uncertainty using a Semantic Gaussian Map, integrating these uncertainties into a 3D Value Map to improve navigation reliability. The approach achieves state-of-the-art results on multiple VLN benchmarks, with improvements of 2-4% in success rate and 3-5% in path length weighted success rate over prior methods.
Vision-Language Navigation (VLN) requires an agent to navigate 3D environments following natural language instructions. During navigation, existing agents commonly encounter perceptual uncertainty, such as insufficient evidence for reliable grounding or ambiguity in interpreting spatial cues, yet they typically ignore such information when predicting actions. In this work, we explicitly model three forms of perceptual uncertainty (i.e., geometric, semantic, and appearance uncertainty) and integrate them into the agent's observation space to enable informed decision-making. Concretely, our agent first constructs a Semantic Gaussian Map (SGM), composed of differentiable 3D Gaussian primitives initialized from panoramic observations, that encodes both the geometric structure and semantic content of the environment. On top of SGM, geometric uncertainty is estimated through variational perturbations of Gaussian position and scale to assess structural reliability; semantic uncertainty is captured by perturbing Gaussian semantic attributes to reveal ambiguous interpretations; and appearance uncertainty is characterized by Fisher Information, which measures the sensitivity of rendered observations to Gaussian-level variations. These uncertainties are incorporated into SGM, extending it into a unified 3D Value Map, which grounds them as affordances and constraints that support reliable navigation. Comprehensive evaluations across multiple VLN benchmarks show the effectiveness of our agent.