Translate-Distill: Learning Cross-Language Dense Retrieval by
  Translation and Distillation
Eugene Yang Dawn Lawrie James Mayfield Douglas W. Oard Scott Miller
Prior work on English monolingual retrieval has shown that a cross-encoder
trained using a large number of relevance judgments for query-document pairs
can be used as a teacher to train more efficient, but similarly effective,
dual-encoder student models. Applying a similar knowledge distillation approach
to training an efficient dual-encoder model for Cross-Language Information
Retrieval (CLIR), where queries and documents are in different languages, is
challenging due to the lack of a sufficiently large training collection when
the query and document languages differ. The state of the art for CLIR thus
relies on translating queries, documents, or both from the large English MS
MARCO training set, an approach called Translate-Train. This paper proposes an
alternative, Translate-Distill, in which knowledge distillation from either a
monolingual cross-encoder or a CLIR cross-encoder is used to train a
dual-encoder CLIR student model. This richer design space enables the teacher
model to perform inference in an optimized setting, while training the student
model directly for CLIR. Trained models and artifacts are publicly available on
Huggingface.