Understanding Gaussian Attention Bias of Vision Transformers Using
  Effective Receptive Fields
Bum Jun Kim Hyeyeon Choi Hyeonah Jang Sang Woo Kim
Vision transformers (ViTs) that model an image as a sequence of partitioned
patches have shown notable performance in diverse vision tasks. Because
partitioning patches eliminates the image structure, to reflect the order of
patches, ViTs utilize an explicit component called positional embedding.
However, we claim that the use of positional embedding does not simply
guarantee the order-awareness of ViT. To support this claim, we analyze the
actual behavior of ViTs using an effective receptive field. We demonstrate that
during training, ViT acquires an understanding of patch order from the
positional embedding that is trained to be a specific pattern. Based on this
observation, we propose explicitly adding a Gaussian attention bias that guides
the positional embedding to have the corresponding pattern from the beginning
of training. We evaluated the influence of Gaussian attention bias on the
performance of ViTs in several image classification, object detection, and
semantic segmentation experiments. The results showed that proposed method not
only facilitates ViTs to understand images but also boosts their performance on
various datasets, including ImageNet, COCO 2017, and ADE20K.