Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal
  Retrieval
Yabing Wang Shuhui Wang Hao Luo Jianfeng Dong Fan Wang Meng Han Xun Wang Meng Wang
Current research on cross-modal retrieval is mostly English-oriented, as the
availability of a large number of English-oriented human-labeled
vision-language corpora. In order to break the limit of non-English labeled
data, cross-lingual cross-modal retrieval (CCR) has attracted increasing
attention. Most CCR methods construct pseudo-parallel vision-language corpora
via Machine Translation (MT) to achieve cross-lingual transfer. However, the
translated sentences from MT are generally imperfect in describing the
corresponding visual contents. Improperly assuming the pseudo-parallel data are
correctly correlated will make the networks overfit to the noisy
correspondence. Therefore, we propose Dual-view Curricular Optimal Transport
(DCOT) to learn with noisy correspondence in CCR. In particular, we quantify
the confidence of the sample pair correlation with optimal transport theory
from both the cross-lingual and cross-modal views, and design dual-view
curriculum learning to dynamically model the transportation costs according to
the learning stage of the two views. Extensive experiments are conducted on two
multilingual image-text datasets and one video-text dataset, and the results
demonstrate the effectiveness and robustness of the proposed method. Besides,
our proposed method also shows a good expansibility to cross-lingual image-text
baselines and a decent generalization on out-of-domain data.