Towards Deviation-Robust Agent Navigation via Perturbation-Aware
  Contrastive Learning
Bingqian Lin Yanxin Long Yi Zhu Fengda Zhu Xiaodan Liang Qixiang Ye Liang Lin
Vision-and-language navigation (VLN) asks an agent to follow a given language
instruction to navigate through a real 3D environment. Despite significant
advances, conventional VLN agents are trained typically under disturbance-free
environments and may easily fail in real-world scenarios, since they are
unaware of how to deal with various possible disturbances, such as sudden
obstacles or human interruptions, which widely exist and may usually cause an
unexpected route deviation. In this paper, we present a model-agnostic training
paradigm, called Progressive Perturbation-aware Contrastive Learning (PROPER)
to enhance the generalization ability of existing VLN agents, by requiring them
to learn towards deviation-robust navigation. Specifically, a simple yet
effective path perturbation scheme is introduced to implement the route
deviation, with which the agent is required to still navigate successfully
following the original instruction. Since directly enforcing the agent to learn
perturbed trajectories may lead to inefficient training, a progressively
perturbed trajectory augmentation strategy is designed, where the agent can
self-adaptively learn to navigate under perturbation with the improvement of
its navigation performance for each specific trajectory. For encouraging the
agent to well capture the difference brought by perturbation, a
perturbation-aware contrastive learning mechanism is further developed by
contrasting perturbation-free trajectory encodings and perturbation-based
counterparts. Extensive experiments on R2R show that PROPER can benefit
multiple VLN baselines in perturbation-free scenarios. We further collect the
perturbed path data to construct an introspection subset based on the R2R,
called Path-Perturbed R2R (PP-R2R). The results on PP-R2R show unsatisfying
robustness of popular VLN agents and the capability of PROPER in improving the
navigation robustness.