Prioritized Semantic Learning for Zero-shot Instance Navigation
This addresses the issue of semantic neglect in navigation agents for robotics or AI systems, offering a significant performance improvement, though it appears incremental as it builds on prior image-goal navigation approaches.
The paper tackles the problem of zero-shot instance navigation, where an agent must navigate to a specific object without training on object annotations, by proposing a Prioritized Semantic Learning (PSL) method to improve semantic understanding. The result is that the PSL agent outperforms the previous state-of-the-art by 66% in success rate on zero-shot ObjectNav and shows superiority on a new InstanceNav task.
We study zero-shot instance navigation, in which the agent navigates to a specific object without using object annotations for training. Previous object navigation approaches apply the image-goal navigation (ImageNav) task (go to the location of an image) for pretraining, and transfer the agent to achieve object goals using a vision-language model. However, these approaches lead to issues of semantic neglect, where the model fails to learn meaningful semantic alignments. In this paper, we propose a Prioritized Semantic Learning (PSL) method to improve the semantic understanding ability of navigation agents. Specifically, a semantic-enhanced PSL agent is proposed and a prioritized semantic training strategy is introduced to select goal images that exhibit clear semantic supervision and relax the reward function from strict exact view matching. At inference time, a semantic expansion inference scheme is designed to preserve the same granularity level of the goal semantic as training. Furthermore, for the popular HM3D environment, we present an Instance Navigation (InstanceNav) task that requires going to a specific object instance with detailed descriptions, as opposed to the Object Navigation (ObjectNav) task where the goal is defined merely by the object category. Our PSL agent outperforms the previous state-of-the-art by 66% on zero-shot ObjectNav in terms of success rate and is also superior on the new InstanceNav task. Code will be released at https://github.com/XinyuSun/PSL-InstanceNav.