Masked Path Modeling for Vision-and-Language Navigation
Zi-Yi Dou Feng Gao Nanyun Peng
Vision-and-language navigation (VLN) agents are trained to navigate in
real-world environments by following natural language instructions. A major
challenge in VLN is the limited availability of training data, which hinders
the models' ability to generalize effectively. Previous approaches have
attempted to address this issue by introducing additional supervision during
training, often requiring costly human-annotated data that restricts
scalability. In this paper, we introduce a masked path modeling (MPM)
objective, which pretrains an agent using self-collected data for downstream
navigation tasks. Our proposed method involves allowing the agent to actively
explore navigation environments without a specific goal and collect the paths
it traverses. Subsequently, we train the agent on this collected data to
reconstruct the original path given a randomly masked subpath. This way, the
agent can actively accumulate a diverse and substantial amount of data while
learning conditional action generation. To evaluate the effectiveness of our
technique, we conduct experiments on various VLN datasets and demonstrate the
versatility of MPM across different levels of instruction complexity. Our
results exhibit significant improvements in success rates, with enhancements of
1.32\%, 1.05\%, and 1.19\% on the val-unseen split of the Room-to-Room,
Room-for-Room, and Room-across-Room datasets, respectively. Furthermore, we
conduct an analysis that highlights the potential for additional improvements
when the agent is allowed to explore unseen environments prior to testing.