Observation, Analysis, and Solution: Exploring Strong Lightweight Vision
  Transformers via Masked Image Modeling Pre-Training
Jin Gao Shubo Lin Shaoru Wang Yutong Kou Zeming Li Liang Li Congxuan Zhang Xiaoqin Zhang Yizheng Wang Weiming Hu
Masked image modeling (MIM) pre-training for large-scale vision transformers
(ViTs) in computer vision has enabled promising downstream performance on top
of the learned self-supervised ViT features. In this paper, we question if the
extremely simple ViTs' fine-tuning performance with a small-scale architecture
can also benefit from this pre-training paradigm, which is considerably less
studied yet in contrast to the well-established lightweight architecture design
methodology with sophisticated components introduced. By carefully adapting
various typical MIM pre-training methods to this lightweight regime and
comparing them with the contrastive learning (CL) pre-training on various
downstream image classification and dense prediction tasks, we systematically
observe different behaviors between MIM and CL with respect to the downstream
fine-tuning data scales. Furthermore, we analyze the frozen features under
linear probing evaluation and also the layer representation similarities and
attention maps across the obtained models, which clearly show the inferior
learning of MIM pre-training on higher layers, leading to unsatisfactory
fine-tuning performance on data-insufficient downstream tasks. This finding is
naturally a guide to choosing appropriate distillation strategies during
pre-training to solve the above deterioration problem. Extensive experiments on
various vision tasks demonstrate the effectiveness of our
observation-analysis-solution flow. In particular, our pre-training with
distillation on pure lightweight ViTs with vanilla/hierarchical design
(5.7M/6.5M) can achieve 79.4%/78.9% top-1 accuracy on ImageNet-1K. It also
enables SOTA performance on the ADE20K semantic segmentation task (42.8% mIoU)
and LaSOT visual tracking task (66.1% AUC) in the lightweight regime. The
latter even surpasses all the current SOTA lightweight CPU-realtime trackers.