RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large
  Language Models
Ronak Pradeep Sahel Sharifymoghaddam Jimmy Lin
Researchers have successfully applied large language models (LLMs) such as
ChatGPT to reranking in an information retrieval context, but to date, such
work has mostly been built on proprietary models hidden behind opaque API
endpoints. This approach yields experimental results that are not reproducible
and non-deterministic, threatening the veracity of outcomes that build on such
shaky foundations. To address this significant shortcoming, we present
RankVicuna, the first fully open-source LLM capable of performing high-quality
listwise reranking in a zero-shot setting. Experimental results on the TREC
2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness
comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter
model, although our effectiveness remains slightly behind reranking with GPT-4.
We hope our work provides the foundation for future research on reranking with
modern LLMs. All the code necessary to reproduce our results is available at
https://github.com/castorini/rank_llm.