Spatial and Visual Perspective-Taking via View Rotation and Relation
  Reasoning for Embodied Reference Understanding
Cheng Shi Sibei Yang
Embodied Reference Understanding studies the reference understanding in an
embodied fashion, where a receiver is required to locate a target object
referred to by both language and gesture of the sender in a shared physical
environment. Its main challenge lies in how to make the receiver with the
egocentric view access spatial and visual information relative to the sender to
judge how objects are oriented around and seen from the sender, i.e., spatial
and visual perspective-taking. In this paper, we propose a REasoning from your
Perspective (REP) method to tackle the challenge by modeling relations between
the receiver and the sender and the sender and the objects via the proposed
novel view rotation and relation reasoning. Specifically, view rotation first
rotates the receiver to the position of the sender by constructing an embodied
3D coordinate system with the position of the sender as the origin. Then, it
changes the orientation of the receiver to the orientation of the sender by
encoding the body orientation and gesture of the sender. Relation reasoning
models the nonverbal and verbal relations between the sender and the objects by
multi-modal cooperative reasoning in gesture, language, visual content, and
spatial position. Experiment results demonstrate the effectiveness of REP,
which consistently surpasses all existing state-of-the-art algorithms by a
large margin, i.e., +5.22% absolute accuracy in terms of Prec0.5 on YouRefIt.