Unsupervised Dense Retrieval with Relevance-Aware Contrastive
  Pre-Training
Yibin Lei Liang Ding Yu Cao Changtong Zan Andrew Yates Dacheng Tao
Dense retrievers have achieved impressive performance, but their demand for
abundant training data limits their application scenarios. Contrastive
pre-training, which constructs pseudo-positive examples from unlabeled data,
has shown great potential to solve this problem. However, the pseudo-positive
examples crafted by data augmentations can be irrelevant. To this end, we
propose relevance-aware contrastive learning. It takes the intermediate-trained
model itself as an imperfect oracle to estimate the relevance of positive pairs
and adaptively weighs the contrastive loss of different pairs according to the
estimated relevance. Our method consistently improves the SOTA unsupervised
Contriever model on the BEIR and open-domain QA retrieval benchmarks. Further
exploration shows that our method can not only beat BM25 after further
pre-training on the target corpus but also serves as a good few-shot learner.
Our code is publicly available at https://github.com/Yibin-Lei/ReContriever.