Delving Deeper into Data Scaling in Masked Image Modeling
Cheng-Ze Lu Xiaojie Jin Qibin Hou Jun Hao Liew Ming-Ming Cheng Jiashi Feng
Understanding whether self-supervised learning methods can scale with
unlimited data is crucial for training large-scale models. In this work, we
conduct an empirical study on the scaling capability of masked image modeling
(MIM) methods (e.g., MAE) for visual recognition. Unlike most previous works
that depend on the widely-used ImageNet dataset, which is manually curated and
object-centric, we take a step further and propose to investigate this problem
in a more practical setting. Specifically, we utilize the web-collected
Coyo-700M dataset. We randomly sample varying numbers of training images from
the Coyo dataset and construct a series of sub-datasets, containing 0.5M, 1M,
5M, 10M, and 100M images, for pre-training. Our goal is to investigate how the
performance changes on downstream tasks when scaling with different sizes of
data and models. The study reveals that: 1) MIM can be viewed as an effective
method to improve the model capacity when the scale of the training data is
relatively small; 2) Strong reconstruction targets can endow the models with
increased capacities on downstream tasks; 3) MIM pre-training is data-agnostic
under most scenarios, which means that the strategy of sampling pre-training
data is non-critical. We hope these observations could provide valuable
insights for future research on MIM.