Building Category Graphs Representation with Spatial and Temporal
  Attention for Visual Navigation
Xiaobo Hu Youfang Lin HeHe Fan Shuo Wang Zhihao Wu Kai Lv
Given an object of interest, visual navigation aims to reach the object's
location based on a sequence of partial observations. To this end, an agent
needs to 1) learn a piece of certain knowledge about the relations of object
categories in the world during training and 2) look for the target object based
on the pre-learned object category relations and its moving trajectory in the
current unseen environment. In this paper, we propose a Category Relation Graph
(CRG) to learn the knowledge of object category layout relations and a
Temporal-Spatial-Region (TSR) attention architecture to perceive the long-term
spatial-temporal dependencies of objects helping the navigation. We learn prior
knowledge of object layout, establishing a category relationship graph to
deduce the positions of specific objects. Subsequently, we introduced TSR to
capture the relationships of objects in temporal, spatial, and regions within
the observation trajectories. Specifically, we propose a Temporal attention
module (T) to model the temporal structure of the observation sequence, which
implicitly encodes the historical moving or trajectory information. Then, a
Spatial attention module (S) is used to uncover the spatial context of the
current observation objects based on the category relation graph and past
observations. Last, a Region attention module (R) shifts the attention to the
target-relevant region. Based on the visual representation extracted by our
method, the agent can better perceive the environment and easily learn superior
navigation policy. Experiments on AI2-THOR demonstrate our CRG-TSR method
significantly outperforms existing methods regarding both effectiveness and
efficiency. The code has been included in the supplementary material and will
be publicly available.