Prioritized Semantic Learning for Zero-shot Instance Navigation
Xander Sun Louis Lau Hoyard Zhi Ronghe Qiu Junwei Liang
We study zero-shot instance navigation, in which the agent navigates to a
specific object without using object annotations for training. Previous object
navigation approaches apply the image-goal navigation (ImageNav) task (go to
the location of an image) for pretraining, and transfer the agent to achieve
object goals using a vision-language model. However, these approaches lead to
issues of semantic neglect, where the model fails to learn meaningful semantic
alignments. In this paper, we propose a Prioritized Semantic Learning (PSL)
method to improve the semantic understanding ability of navigation agents.
Specifically, a semantic-enhanced PSL agent is proposed and a prioritized
semantic training strategy is introduced to select goal images that exhibit
clear semantic supervision and relax the reward function from strict exact view
matching. At inference time, a semantic expansion inference scheme is designed
to preserve the same granularity level of the goal-semantic as training.
Furthermore, for the popular HM3D environment, we present an Instance
Navigation (InstanceNav) task that requires going to a specific object instance
with detailed descriptions, as opposed to the Object Navigation (ObjectNav)
task where the goal is defined merely by the object category. Our PSL agent
outperforms the previous state-of-the-art by 66% on zero-shot ObjectNav in
terms of success rate and is also superior on the new InstanceNav task. Code
will be released at https://anonymous.4open. science/r/PSL/.