Interaction-aware Joint Attention Estimation Using People Attributes
Chihiro Nakatani Hiroaki Kawashima Norimichi Ukita
This paper proposes joint attention estimation in a single image. Different
from related work in which only the gaze-related attributes of people are
independently employed, (I) their locations and actions are also employed as
contextual cues for weighting their attributes, and (ii) interactions among all
of these attributes are explicitly modeled in our method. For the interaction
modeling, we propose a novel Transformer-based attention network to encode
joint attention as low-dimensional features. We introduce a specialized MLP
head with positional embedding to the Transformer so that it predicts pixelwise
confidence of joint attention for generating the confidence heatmap. This
pixelwise prediction improves the heatmap accuracy by avoiding the ill-posed
problem in which the high-dimensional heatmap is predicted from the
low-dimensional features. The estimated joint attention is further improved by
being integrated with general image-based attention estimation. Our method
outperforms SOTA methods quantitatively in comparative experiments. Code:
https://anonymous.4open.science/r/anonymized_codes-ECA4.