Learning Vision-and-Language Navigation from YouTube Videos
Kunyang Lin Peihao Chen Diwei Huang Thomas H. Li Mingkui Tan Chuang Gan
Vision-and-language navigation (VLN) requires an embodied agent to navigate
in realistic 3D environments using natural language instructions. Existing VLN
methods suffer from training on small-scale environments or unreasonable
path-instruction datasets, limiting the generalization to unseen environments.
There are massive house tour videos on YouTube, providing abundant real
navigation experiences and layout information. However, these videos have not
been explored for VLN before. In this paper, we propose to learn an agent from
these videos by creating a large-scale dataset which comprises reasonable
path-instruction pairs from house tour videos and pre-training the agent on it.
To achieve this, we have to tackle the challenges of automatically constructing
path-instruction pairs and exploiting real layout knowledge from raw and
unlabeled videos. To address these, we first leverage an entropy-based method
to construct the nodes of a path trajectory. Then, we propose an action-aware
generator for generating instructions from unlabeled trajectories. Last, we
devise a trajectory judgment pretext task to encourage the agent to mine the
layout knowledge. Experimental results show that our method achieves
state-of-the-art performance on two popular benchmarks (R2R and REVERIE). Code
is available at https://github.com/JeremyLinky/YouTube-VLN