Fuse after Align: Improving Face-Voice Association Learning via
  Multimodal Encoder
Chong Peng Liqiang He Dan Su
Today, there have been many achievements in learning the association between
voice and face. However, most previous work models rely on cosine similarity or
L2 distance to evaluate the likeness of voices and faces following contrastive
learning, subsequently applied to retrieval and matching tasks. This method
only considers the embeddings as high-dimensional vectors, utilizing a minimal
scope of available information. This paper introduces a novel framework within
an unsupervised setting for learning voice-face associations. By employing a
multimodal encoder after contrastive learning and addressing the problem
through binary classification, we can learn the implicit information within the
embeddings in a more effective and varied manner. Furthermore, by introducing
an effective pair selection method, we enhance the learning outcomes of both
contrastive learning and the matching task. Empirical evidence demonstrates
that our framework achieves state-of-the-art results in voice-face matching,
verification, and retrieval tasks, improving verification by approximately 3%,
matching by about 2.5%, and retrieval by around 1.3%.