BERM: Training the Balanced and Extractable Representation for Matching
  to Improve Generalization Ability of Dense Retrieval
Shicheng Xu Liang Pang Huawei Shen Xueqi Cheng
Dense retrieval has shown promise in the first-stage retrieval process when
trained on in-domain labeled datasets. However, previous studies have found
that dense retrieval is hard to generalize to unseen domains due to its weak
modeling of domain-invariant and interpretable feature (i.e., matching signal
between two texts, which is the essence of information retrieval). In this
paper, we propose a novel method to improve the generalization of dense
retrieval via capturing matching signal called BERM. Fully fine-grained
expression and query-oriented saliency are two properties of the matching
signal. Thus, in BERM, a single passage is segmented into multiple units and
two unit-level requirements are proposed for representation as the constraint
in training to obtain the effective matching signal. One is semantic unit
balance and the other is essential matching unit extractability. Unit-level
view and balanced semantics make representation express the text in a
fine-grained manner. Essential matching unit extractability makes passage
representation sensitive to the given query to extract the pure matching
information from the passage containing complex context. Experiments on BEIR
show that our method can be effectively combined with different dense retrieval
training methods (vanilla, hard negatives mining and knowledge distillation) to
improve its generalization ability without any additional inference overhead
and target domain data.