Text-Guided Face Recognition using Multi-Granularity Cross-Modal
  Contrastive Learning
Md Mahedi Hasan Shoaib Meraj Sami Nasser Nasrabadi
State-of-the-art face recognition (FR) models often experience a significant
performance drop when dealing with facial images in surveillance scenarios
where images are in low quality and often corrupted with noise. Leveraging
facial characteristics, such as freckles, scars, gender, and ethnicity, becomes
highly beneficial in improving FR performance in such scenarios. In this paper,
we introduce text-guided face recognition (TGFR) to analyze the impact of
integrating facial attributes in the form of natural language descriptions. We
hypothesize that adding semantic information into the loop can significantly
improve the image understanding capability of an FR algorithm compared to other
soft biometrics. However, learning a discriminative joint embedding within the
multimodal space poses a considerable challenge due to the semantic gap in the
unaligned image-text representations, along with the complexities arising from
ambiguous and incoherent textual descriptions of the face. To address these
challenges, we introduce a face-caption alignment module (FCAM), which
incorporates cross-modal contrastive losses across multiple granularities to
maximize the mutual information between local and global features of the
face-caption pair. Within FCAM, we refine both facial and textual features for
learning aligned and discriminative features. We also design a face-caption
fusion module (FCFM) that applies fine-grained interactions and coarse-grained
associations among cross-modal features. Through extensive experiments
conducted on three face-caption datasets, proposed TGFR demonstrates remarkable
improvements, particularly on low-quality images, over existing FR models and
outperforms other related methods and benchmarks.