Eye vs. AI: Human Gaze and Model Attention in Video Memorability
Prajneya Kumar Eshika Khandelwal Makarand Tapaswi Vishnu Sreekumar
Understanding the factors that determine video memorability has important
applications in areas such as educational technology and advertising. Towards
this goal, we investigate the semantic and temporal attention mechanisms
underlying video memorability. We propose a Transformer-based model with
spatio-temporal attention that matches SoTA performance on video memorability
prediction on a large naturalistic video dataset. More importantly, the
self-attention patterns show us where the model looks to predict memorability.
We compare model attention against human gaze fixation density maps collected
through a small-scale eye-tracking experiment where humans perform a video
memory task. Quantitative saliency metrics show that the model attention and
human gaze follow similar patterns. Furthermore, while panoptic segmentation
confirms that the model and humans attend more to thing classes, stuff classes
that receive increased/decreased attention tend to have higher memorability
scores. We also observe that the model assigns greater importance to the
initial frames, mimicking temporal attention patterns found in humans.