Bootstrap Masked Visual Modeling via Hard Patches Mining
Haochen Wang Junsong Fan Yuxi Wang Kaiyou Song Tiancai Wang Xiangyu Zhang Zhaoxiang Zhang
Masked visual modeling has attracted much attention due to its promising
potential in learning generalizable representations. Typical approaches urge
models to predict specific contents of masked tokens, which can be intuitively
considered as teaching a student (the model) to solve given problems
(predicting masked contents). Under such settings, the performance is highly
correlated with mask strategies (the difficulty of provided problems). We argue
that it is equally important for the model to stand in the shoes of a teacher
to produce challenging problems by itself. Intuitively, patches with high
values of reconstruction loss can be regarded as hard samples, and masking
those hard patches naturally becomes a demanding reconstruction task. To
empower the model as a teacher, we propose Hard Patches Mining (HPM),
predicting patch-wise losses and subsequently determining where to mask.
Technically, we introduce an auxiliary loss predictor, which is trained with a
relative objective to prevent overfitting to exact loss values. Also, to
gradually guide the training procedure, we propose an easy-to-hard mask
strategy. Empirically, HPM brings significant improvements under both image and
video benchmarks. Interestingly, solely incorporating the extra loss prediction
objective leads to better representations, verifying the efficacy of
determining where is hard to reconstruct. The code is available at
https://github.com/Haochen-Wang409/HPM.