JaColBERT and Hard Negatives, Towards Better Japanese-First Embeddings
  for Retrieval: Early Technical Report
Benjamin Clavié
Document retrieval in many languages has been largely relying on
multi-lingual models, and leveraging the vast wealth of English training data.
In Japanese, the best performing deep-learning based retrieval approaches rely
on multilingual dense embeddings. In this work, we introduce (1) a
hard-negative augmented version of the Japanese MMARCO dataset and (2)
JaColBERT, a document retrieval model built on the ColBERT model architecture,
specifically for Japanese. JaColBERT vastly outperform all previous monolingual
retrieval approaches and competes with the best multilingual methods, despite
unfavourable evaluation settings (out-of-domain vs. in-domain for the
multilingual models). JaColBERT reaches an average Recall@10 of 0.813,
noticeably ahead of the previous monolingual best-performing model (0.716) and
only slightly behind multilingual-e5-base (0.820), though more noticeably
behind multilingual-e5-large (0.856). These results are achieved using only a
limited, entirely Japanese, training set, more than two orders of magnitudes
smaller than multilingual embedding models. We believe these results show great
promise to support retrieval-enhanced application pipelines in a wide variety
of domains.