Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention
This addresses the challenge of enabling AI agents to navigate complex environments with limited human guidance, which is incremental by building on imitation learning with a novel intervention framework.
The paper tackles the problem of vision-based navigation with language assistance, where an agent finds objects in indoor environments using high-level goals and queries an advisor for subgoals when lost, resulting in a significant improvement in success rates over baselines in both seen and unseen environments.
We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level end-goals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines in both seen and unseen environments. Our code and data are publicly available at https://github.com/debadeepta/vnla .