Semi-supervised Semantic Segmentation Meets Masked Modeling:Fine-grained
  Locality Learning Matters in Consistency Regularization
Wentao Pan Zhe Xu Jiangpeng Yan Zihan Wu Raymond Kai-yu Tong Xiu Li Jianhua Yao
Semi-supervised semantic segmentation aims to utilize limited labeled images
and abundant unlabeled images to achieve label-efficient learning, wherein the
weak-to-strong consistency regularization framework, popularized by FixMatch,
is widely used as a benchmark scheme. Despite its effectiveness, we observe
that such scheme struggles with satisfactory segmentation for the local
regions. This can be because it originally stems from the image classification
task and lacks specialized mechanisms to capture fine-grained local semantics
that prioritizes in dense prediction. To address this issue, we propose a novel
framework called \texttt{MaskMatch}, which enables fine-grained locality
learning to achieve better dense segmentation. On top of the original
teacher-student framework, we design a masked modeling proxy task that
encourages the student model to predict the segmentation given the unmasked
image patches (even with 30\% only) and enforces the predictions to be
consistent with pseudo-labels generated by the teacher model using the complete
image. Such design is motivated by the intuition that if the predictions are
more consistent given insufficient neighboring information, stronger
fine-grained locality perception is achieved. Besides, recognizing the
importance of reliable pseudo-labels in the above locality learning and the
original consistency learning scheme, we design a multi-scale ensembling
strategy that considers context at different levels of abstraction for
pseudo-label generation. Extensive experiments on benchmark datasets
demonstrate the superiority of our method against previous approaches and its
plug-and-play flexibility.