OVER-NAV: Elevating Iterative Vision-and-Language Navigation with
  Open-Vocabulary Detection and StructurEd Representation
Ganlong Zhao Guanbin Li Weikai Chen Yizhou Yu
Recent advances in Iterative Vision-and-Language Navigation (IVLN) introduce
a more meaningful and practical paradigm of VLN by maintaining the agent's
memory across tours of scenes. Although the long-term memory aligns better with
the persistent nature of the VLN task, it poses more challenges on how to
utilize the highly unstructured navigation memory with extremely sparse
supervision. Towards this end, we propose OVER-NAV, which aims to go over and
beyond the current arts of IVLN techniques. In particular, we propose to
incorporate LLMs and open-vocabulary detectors to distill key information and
establish correspondence between multi-modal signals. Such a mechanism
introduces reliable cross-modal supervision and enables on-the-fly
generalization to unseen scenes without the need of extra annotation and
re-training. To fully exploit the interpreted navigation data, we further
introduce a structured representation, coded Omnigraph, to effectively
integrate multi-modal information along the tour. Accompanied with a novel
omnigraph fusion mechanism, OVER-NAV is able to extract the most relevant
knowledge from omnigraph for a more accurate navigating action. In addition,
OVER-NAV seamlessly supports both discrete and continuous environments under a
unified framework. We demonstrate the superiority of OVER-NAV in extensive
experiments.