VLN-Trans: Translator for the Vision and Language Navigation Agent
This addresses navigation challenges for AI agents in real-world environments where human instructions may be unclear, though it appears incremental as it builds on existing navigation frameworks.
The paper tackles the problem of vision-and-language navigation agents struggling with ambiguous or unrecognizable landmarks in instructions by designing a translator module that converts original instructions into easy-to-follow sub-instructions, achieving state-of-the-art results on R2R, R4R, and R2R-Last datasets.
Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent's visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room~(R2R), Room4room~(R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.