Semantic-Aware Autoregressive Image Modeling for Visual Representation
  Learning
Kaiyou Song Shan Zhang Tong Wang
The development of autoregressive modeling (AM) in computer vision lags
behind natural language processing (NLP) in self-supervised pre-training. This
is mainly caused by the challenge that images are not sequential signals and
lack a natural order when applying autoregressive modeling. In this study,
inspired by human beings' way of grasping an image, i.e., focusing on the main
object first, we present a semantic-aware autoregressive image modeling
(SemAIM) method to tackle this challenge. The key insight of SemAIM is to
autoregressive model images from the semantic patches to the less semantic
patches. To this end, we first calculate a semantic-aware permutation of
patches according to their feature similarities and then perform the
autoregression procedure based on the permutation. In addition, considering
that the raw pixels of patches are low-level signals and are not ideal
prediction targets for learning high-level semantic representation, we also
explore utilizing the patch features as the prediction targets. Extensive
experiments are conducted on a broad range of downstream tasks, including image
classification, object detection, and instance/semantic segmentation, to
evaluate the performance of SemAIM. The results demonstrate SemAIM achieves
state-of-the-art performance compared with other self-supervised methods.
Specifically, with ViT-B, SemAIM achieves 84.1% top-1 accuracy for fine-tuning
on ImageNet, 51.3% AP and 45.4% AP for object detection and instance
segmentation on COCO, which outperforms the vanilla MAE by 0.5%, 1.0%, and
0.5%, respectively.