Resources for Brewing BEIR: Reproducible Reference Models and an
  Official Leaderboard
Ehsan Kamalloo Nandan Thakur Carlos Lassance Xueguang Ma Jheng-Hong Yang Jimmy Lin
BEIR is a benchmark dataset for zero-shot evaluation of information retrieval
models across 18 different domain/task combinations. In recent years, we have
witnessed the growing popularity of a representation learning approach to
building retrieval models, typically using pretrained transformers in a
supervised setting. This naturally begs the question: How effective are these
models when presented with queries and documents that differ from the training
data? Examples include searching in different domains (e.g., medical or legal
text) and with different types of queries (e.g., keywords vs. well-formed
questions). While BEIR was designed to answer these questions, our work
addresses two shortcomings that prevent the benchmark from achieving its full
potential: First, the sophistication of modern neural methods and the
complexity of current software infrastructure create barriers to entry for
newcomers. To this end, we provide reproducible reference implementations that
cover the two main classes of approaches: learned dense and sparse models.
Second, there does not exist a single authoritative nexus for reporting the
effectiveness of different models on BEIR, which has led to difficulty in
comparing different methods. To remedy this, we present an official
self-service BEIR leaderboard that provides fair and consistent comparisons of
retrieval models. By addressing both shortcomings, our work facilitates future
explorations in a range of interesting research questions that BEIR enables.