PRD: Peer Rank and Discussion Improve Large Language Model based
  Evaluations
Ruosen Li Teerth Patel Xinya Du
Nowadays, the quality of responses generated by different modern large
language models (LLMs) are hard to evaluate and compare automatically. Recent
studies suggest and predominantly use LLMs as a reference-free metric for
open-ended question answering. More specifically, they use the recognized
"strongest" LLM as the evaluator, which conducts pairwise comparisons of
candidate models' answers and provides a ranking score. However, this intuitive
method has multiple problems, such as bringing in self-enhancement (favoring
its own answers) and positional bias. We draw insights and lessons from the
educational domain (Cho and MacArthur, 2011; Walsh, 2014) to improve LLM-based
evaluations. Specifically, we propose the (1) peer rank (PR) algorithm that
takes into account each peer LLM's pairwise preferences of all answer pairs,
and outputs a final ranking of models; and (2) peer discussion (PD), where we
prompt two LLMs to discuss and try to reach a mutual agreement on preferences
of two answers. We conduct experiments on two benchmark datasets. We find that
our approaches achieve higher accuracy and align better with human judgments,
respectively. Interestingly, PR can induce a relatively accurate self-ranking
of models under the anonymous setting, where each model's name is unrevealed.
Our work provides space to explore evaluating models that are hard to compare
for humans.