A Modular Multimodal Architecture for Gaze Target Prediction:
  Application to Privacy-Sensitive Settings
Anshul Gupta Samy Tafasca Jean-Marc Odobez
Predicting where a person is looking is a complex task, requiring to
understand not only the person's gaze and scene content, but also the 3D scene
structure and the person's situation (are they manipulating? interacting or
observing others? attentive?) to detect obstructions in the line of sight or
apply attention priors that humans typically have when observing others. In
this paper, we hypothesize that identifying and leveraging such priors can be
better achieved through the exploitation of explicitly derived multimodal cues
such as depth and pose. We thus propose a modular multimodal architecture
allowing to combine these cues using an attention mechanism. The architecture
can naturally be exploited in privacy-sensitive situations such as surveillance
and health, where personally identifiable information cannot be released. We
perform extensive experiments on the GazeFollow and VideoAttentionTarget public
datasets, obtaining state-of-the-art performance and demonstrating very
competitive results in the privacy setting case.