Diagnosing the Environment Bias in Vision-and-Language Navigation
This addresses a major challenge in VLN research by diagnosing and mitigating environment bias, which is crucial for deploying agents in real-world, novel settings, though it is incremental as it builds on existing methods without new model architectures.
The paper tackles the environment bias problem in Vision-and-Language Navigation (VLN), where agents perform poorly in unseen environments due to overfitting on low-level visual features from training data; by replacing these with semantic representations, it reduces performance gaps between seen and unseen environments on multiple datasets, achieving competitive unseen results without modifying the baseline model.
Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are crucial when the agent is navigating new environments about which it has no prior knowledge. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations that contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gaps between seen and unseen on multiple datasets (i.e. R2R, R4R, and CVDN) and achieve competitive unseen results to previous state-of-the-art models. Our code and features are available at: https://github.com/zhangybzbo/EnvBiasVLN