Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced
  Pre-training
Hyesong Choi Hyejin Park Kwang Moo Yi Sungmin Cha Dongbo Min
In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel
and cost-effective approach that significantly enhances the pre-training
performance of Masked Image Modeling (MIM) approaches by prioritizing token
salience. Our method provides robustness against variations in masking ratios,
effectively mitigating the performance instability issues common in existing
methods. This relaxes the sensitivity of MIM-based pre-training to masking
ratios, which in turn allows us to propose an adaptive strategy for `tailored'
masking ratios for each data sample, which no existing method can provide.
Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that
dynamically adjusts the proportion of masking for the unique content of each
image based on token salience. We show that our method significantly improves
over the state-of-the-art in mask-based pre-training on the ImageNet-1K
dataset.