Which way is `right'?: Uncovering limitations of Vision-and-Language
  Navigation model
Meera Hahn Amit Raj James M. Rehg
The challenging task of Vision-and-Language Navigation (VLN) requires
embodied agents to follow natural language instructions to reach a goal
location or object (e.g. `walk down the hallway and turn left at the piano').
For agents to complete this task successfully, they must be able to ground
objects referenced into the instruction (e.g.`piano') into the visual scene as
well as ground directional phrases (e.g.`turn left') into actions. In this work
we ask the following question -- to what degree are spatial and directional
language cues informing the navigation model's decisions? We propose a series
of simple masking experiments to inspect the model's reliance on different
parts of the instruction. Surprisingly we uncover that certain top performing
models rely only on the noun tokens of the instructions. We propose two
training methods to alleviate this concerning limitation.