REPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for
  Noisy Correspondence
Ruochen Zheng Jiahao Hong Changxin Gao Nong Sang
The presence of noise in acquired data invariably leads to performance
degradation in cross-modal matching. Unfortunately, obtaining precise
annotations in the multimodal field is expensive, which has prompted some
methods to tackle the mismatched data pair issue in cross-modal matching
contexts, termed as noisy correspondence. However, most of these existing noisy
correspondence methods exhibit the following limitations: a) the problem of
self-reinforcing error accumulation, and b) improper handling of noisy data
pair. To tackle the two problems, we propose a generalized framework termed as
Rank corrElation and noisy Pair hAlf-replacing wIth memoRy (REPAIR), which
benefits from maintaining a memory bank for features of matched pairs.
Specifically, we calculate the distances between the features in the memory
bank and those of the target pair for each respective modality, and use the
rank correlation of these two sets of distances to estimate the soft
correspondence label of the target pair. Estimating soft correspondence based
on memory bank features rather than using a similarity network can avoid the
accumulation of errors due to incorrect network identifications. For pairs that
are completely mismatched, REPAIR searches the memory bank for the most
matching feature to replace one feature of one modality, instead of using the
original pair directly or merely discarding the mismatched pair. We conduct
experiments on three cross-modal datasets, i.e., Flickr30K, MSCOCO, and CC152K,
proving the effectiveness and robustness of our REPAIR on synthetic and
real-world noise.