Heuristic Vision Pre-Training with Self-Supervised and Supervised
  Multi-Task Learning
Zhiming Qian
To mimic human vision with the way of recognizing the diverse and open world,
foundation vision models are much critical. While recent techniques of
self-supervised learning show the promising potentiality of this mission, we
argue that signals from labelled data are also important for common-sense
recognition, and properly chosen pre-text tasks can facilitate the efficiency
of vision representation learning. To this end, we propose a novel pre-training
framework by adopting both self-supervised and supervised visual pre-text tasks
in a multi-task manner. Specifically, given an image, we take a heuristic way
by considering its intrinsic style properties, inside objects with their
locations and correlations, and how it looks like in 3D space for basic visual
understanding. However, large-scale object bounding boxes and correlations are
usually hard to achieve. Alternatively, we develop a hybrid method by
leveraging both multi-label classification and self-supervised learning. On the
one hand, under the multi-label supervision, the pre-trained model can explore
the detailed information of an image, e.g., image types, objects, and part of
semantic relations. On the other hand, self-supervised learning tasks, with
respect to Masked Image Modeling (MIM) and contrastive learning, can help the
model learn pixel details and patch correlations. Results show that our
pre-trained models can deliver results on par with or better than
state-of-the-art (SOTA) results on multiple visual tasks. For example, with a
vanilla Swin-B backbone, we achieve 85.3\% top-1 accuracy on ImageNet-1K
classification, 47.9 box AP on COCO object detection for Mask R-CNN, and 50.6
mIoU on ADE-20K semantic segmentation when using Upernet. The performance shows
the ability of our vision foundation model to serve general purpose vision
tasks.