Open-source Large Language Models are Strong Zero-shot Query Likelihood
  Models for Document Ranking
Shengyao Zhuang Bing Liu Bevan Koopman Guido Zuccon
In the field of information retrieval, Query Likelihood Models (QLMs) rank
documents based on the probability of generating the query given the content of
a document. Recently, advanced large language models (LLMs) have emerged as
effective QLMs, showcasing promising ranking capabilities. This paper focuses
on investigating the genuine zero-shot ranking effectiveness of recent LLMs,
which are solely pre-trained on unstructured text data without supervised
instruction fine-tuning. Our findings reveal the robust zero-shot ranking
ability of such LLMs, highlighting that additional instruction fine-tuning may
hinder effectiveness unless a question generation task is present in the
fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking
system that integrates LLM-based QLMs with a hybrid zero-shot retriever,
demonstrating exceptional effectiveness in both zero-shot and few-shot
scenarios. We make our codebase publicly available at
https://github.com/ielab/llm-qlm.