ViTGaze: Gaze Following with Interaction Features in Vision Transformers
Yuehao Song Xinggang Wang Jingfeng Yao Wenyu Liu Jinglin Zhang Xiangmin Xu
Gaze following aims to interpret human-scene interactions by predicting the
person's focal point of gaze. Prevailing approaches often use multi-modality
inputs, most of which adopt a two-stage framework. Hence their performance
highly depends on the previous prediction accuracy. Others use a
single-modality approach with complex decoders, increasing network
computational load. Inspired by the remarkable success of pre-trained plain
Vision Transformers (ViTs), we introduce a novel single-modality gaze following
framework, ViTGaze. In contrast to previous methods, ViTGaze creates a brand
new gaze following framework based mainly on powerful encoders (dec. param.
less than 1%). Our principal insight lies in that the inter-token interactions
within self-attention can be transferred to interactions between humans and
scenes. Leveraging this presumption, we formulate a framework consisting of a
4D interaction encoder and a 2D spatial guidance module to extract human-scene
interaction information from self-attention maps. Furthermore, our
investigation reveals that ViT with self-supervised pre-training exhibits an
enhanced ability to extract correlated information. A large number of
experiments have been conducted to demonstrate the performance of the proposed
method. Our method achieves state-of-the-art (SOTA) performance among all
single-modality methods (3.4% improvement on AUC, 5.1% improvement on AP) and
very comparable performance against multi-modality methods with 59% number of
parameters less.