One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones
It addresses the problem of long-horizon task completion for autonomous agents, representing an incremental improvement over existing methods.
The paper tackles the challenge of long-horizon vision-and-language navigation, where agents often fail due to ignoring instructions or getting stuck, by proposing a milestone-based task tracker (M-TRACK) that guides agents step-by-step. This approach achieves a 33% and 52% relative improvement in unseen success rates on the ALFRED dataset.
We study the problem of developing autonomous agents that can follow human instructions to infer and perform a sequence of actions to complete the underlying task. Significant progress has been made in recent years, especially for tasks with short horizons. However, when it comes to long-horizon tasks with extended sequences of actions, an agent can easily ignore some instructions or get stuck in the middle of the long instructions and eventually fail the task. To address this challenge, we propose a model-agnostic milestone-based task tracker (M-TRACK) to guide the agent and monitor its progress. Specifically, we propose a milestone builder that tags the instructions with navigation and interaction milestones which the agent needs to complete step by step, and a milestone checker that systemically checks the agent's progress in its current milestone and determines when to proceed to the next. On the challenging ALFRED dataset, our M-TRACK leads to a notable 33% and 52% relative improvement in unseen success rate over two competitive base models.