Affinity-based Attention in Self-supervised Transformers Predicts
  Dynamics of Object Grouping in Humans
Hossein Adeli Seoyoung Ahn Nikolaus Kriegeskorte Gregory Zelinsky
The spreading of attention has been proposed as a mechanism for how humans
group features to segment objects. However, such a mechanism has not yet been
implemented and tested in naturalistic images. Here, we leverage the feature
maps from self-supervised vision Transformers and propose a model of human
object-based attention spreading and segmentation. Attention spreads within an
object through the feature affinity signal between different patches of the
image. We also collected behavioral data on people grouping objects in natural
images by judging whether two dots are on the same object or on two different
objects. We found that our models of affinity spread that were built on feature
maps from the self-supervised Transformers showed significant improvement over
baseline and CNN based models on predicting reaction time patterns of humans,
despite not being trained on the task or with any other object labels. Our work
provides new benchmarks for evaluating models of visual representation learning
including Transformers.