BiLMa: Bidirectional Local-Matching for Text-based Person
  Re-identification
Takuro Fujii Shuhei Tarashima
Text-based person re-identification (TBPReID) aims to retrieve person images
represented by a given textual query. In this task, how to effectively align
images and texts globally and locally is a crucial challenge. Recent works have
obtained high performances by solving Masked Language Modeling (MLM) to align
image/text parts. However, they only performed uni-directional (i.e., from
image to text) local-matching, leaving room for improvement by introducing
opposite-directional (i.e., from text to image) local-matching. In this work,
we introduce Bidirectional Local-Matching (BiLMa) framework that jointly
optimize MLM and Masked Image Modeling (MIM) in TBPReID model training. With
this framework, our model is trained so as the labels of randomly masked both
image and text tokens are predicted by unmasked tokens. In addition, to narrow
the semantic gap between image and text in MIM, we propose Semantic MIM
(SemMIM), in which the labels of masked image tokens are automatically given by
a state-of-the-art human parser. Experimental results demonstrate that our
BiLMa framework with SemMIM achieves state-of-the-art Rank@1 and mAP scores on
three benchmarks.