Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation
This work addresses a domain-specific issue in VLN by improving visual representation for navigation agents, offering an incremental enhancement through auxiliary pre-training tasks.
The paper tackles the problem of distribution shift between ImageNet pre-trained image encoders and navigation environments in Vision-and-Language Navigation (VLN) by designing structure-encoding auxiliary tasks (SEA) to pre-train the image encoder, resulting in absolute success rate improvements of up to 12% on Test-Unseen environments for various VLN agents.
In Vision-and-Language Navigation (VLN), researchers typically take an image encoder pre-trained on ImageNet without fine-tuning on the environments that the agent will be trained or tested on. However, the distribution shift between the training images from ImageNet and the views in the navigation environments may render the ImageNet pre-trained image encoder suboptimal. Therefore, in this paper, we design a set of structure-encoding auxiliary tasks (SEA) that leverage the data in the navigation environments to pre-train and improve the image encoder. Specifically, we design and customize (1) 3D jigsaw, (2) traversability prediction, and (3) instance classification to pre-train the image encoder. Through rigorous ablations, our SEA pre-trained features are shown to better encode structural information of the scenes, which ImageNet pre-trained features fail to properly encode but is crucial for the target navigation task. The SEA pre-trained features can be easily plugged into existing VLN agents without any tuning. For example, on Test-Unseen environments, the VLN agents combined with our SEA pre-trained features achieve absolute success rate improvement of 12% for Speaker-Follower, 5% for Env-Dropout, and 4% for AuxRN.