Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language
  Pre-training
Haowei Liu Yaya Shi Haiyang Xu Chunfeng Yuan Qinghao Ye Chenliang Li Ming Yan Ji Zhang Fei Huang Bing Li Weiming Hu
In vision-language pre-training (VLP), masked image modeling (MIM) has
recently been introduced for fine-grained cross-modal alignment. However, in
most existing methods, the reconstruction targets for MIM lack high-level
semantics, and text is not sufficiently involved in masked modeling. These two
drawbacks limit the effect of MIM in facilitating cross-modal semantic
alignment. In this work, we propose a semantics-enhanced cross-modal MIM
framework (SemMIM) for vision-language representation learning. Specifically,
to provide more semantically meaningful supervision for MIM, we propose a local
semantics enhancing approach, which harvest high-level semantics from global
image features via self-supervised agreement learning and transfer them to
local patch encodings by sharing the encoding space. Moreover, to achieve deep
involvement of text during the entire MIM process, we propose a text-guided
masking strategy and devise an efficient way of injecting textual information
in both masked modeling and reconstruction target acquisition. Experimental
results validate that our method improves the effectiveness of the MIM task in
facilitating cross-modal semantic alignment. Compared to previous VLP models
with similar model size and data scale, our SemMIM model achieves
state-of-the-art or competitive performance on multiple downstream
vision-language tasks.