Fixating on Attention: Integrating Human Eye Tracking into Vision
  Transformers
Sharath Koorathota Nikolas Papadopoulos Jia Li Ma Shruti Kumar Xiaoxiao Sun Arunesh Mittal Patrick Adelman Paul Sajda
Modern transformer-based models designed for computer vision have
outperformed humans across a spectrum of visual tasks. However, critical tasks,
such as medical image interpretation or autonomous driving, still require
reliance on human judgments. This work demonstrates how human visual input,
specifically fixations collected from an eye-tracking device, can be integrated
into transformer models to improve accuracy across multiple driving situations
and datasets. First, we establish the significance of fixation regions in
left-right driving decisions, as observed in both human subjects and a Vision
Transformer (ViT). By comparing the similarity between human fixation maps and
ViT attention weights, we reveal the dynamics of overlap across individual
heads and layers. This overlap is exploited for model pruning without
compromising accuracy. Thereafter, we incorporate information from the driving
scene with fixation data, employing a "joint space-fixation" (JSF) attention
setup. Lastly, we propose a "fixation-attention intersection" (FAX) loss to
train the ViT model to attend to the same regions that humans fixated on. We
find that the ViT performance is improved in accuracy and number of training
epochs when using JSF and FAX. These results hold significant implications for
human-guided artificial intelligence.