Prompt-based Context- and Domain-aware Pretraining for Vision and Language Navigation
This work addresses the challenge of improving VLN agents' performance in indoor environments, which is incremental as it builds on existing pretrained models with prompt-based tuning.
The paper tackles the problem of pretrained visual-language models lacking sensitivity to indoor scenarios and contextual relations in vision and language navigation (VLN), proposing a prompt-based pretraining framework that achieves state-of-the-art results on R2R and REVERIE benchmarks.
Pretrained visual-language models have extensive world knowledge and are widely used in visual and language navigation (VLN). However, they are not sensitive to indoor scenarios for VLN tasks. Another challenge for VLN is how the agent understands the contextual relations between actions on a path and performs cross-modal alignment sequentially. In this paper, we propose a novel Prompt-bAsed coNtext- and inDoor-Aware (PANDA) pretraining framework to address these problems. It performs prompting in two stages. In the indoor-aware stage, we apply an efficient tuning paradigm to learn deep visual prompts from an indoor dataset, in order to augment pretrained models with inductive biases towards indoor environments. This can enable more sample-efficient adaptation for VLN agents. Furthermore, in the context-aware stage, we design a set of hard context prompts to capture the sequence-level semantics in the instruction. They enable further tuning of the pretrained models via contrastive learning. Experimental results on both R2R and REVERIE show the superiority of PANDA compared to existing state-of-the-art methods.