Masking Improves Contrastive Self-Supervised Learning for ConvNets, and
  Saliency Tells You Where
Zhi-Yi Chin Chieh-Ming Jiang Ching-Chun Huang Pin-Yu Chen Wei-Chen Chiu
While image data starts to enjoy the simple-but-effective self-supervised
learning scheme built upon masking and self-reconstruction objective thanks to
the introduction of tokenization procedure and vision transformer backbone,
convolutional neural networks as another important and widely-adopted
architecture for image data, though having contrastive-learning techniques to
drive the self-supervised learning, still face the difficulty of leveraging
such straightforward and general masking operation to benefit their learning
process significantly. In this work, we aim to alleviate the burden of
including masking operation into the contrastive-learning framework for
convolutional neural networks as an extra augmentation method. In addition to
the additive but unwanted edges (between masked and unmasked regions) as well
as other adverse effects caused by the masking operations for ConvNets, which
have been discussed by prior works, we particularly identify the potential
problem where for one view in a contrastive sample-pair the randomly-sampled
masking regions could be overly concentrated on important/salient objects thus
resulting in misleading contrastiveness to the other view. To this end, we
propose to explicitly take the saliency constraint into consideration in which
the masked regions are more evenly distributed among the foreground and
background for realizing the masking-based augmentation. Moreover, we introduce
hard negative samples by masking larger regions of salient patches in an input
image. Extensive experiments conducted on various datasets, contrastive
learning mechanisms, and downstream tasks well verify the efficacy as well as
the superior performance of our proposed method with respect to several
state-of-the-art baselines.