An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned
  Judge Models are Task-specific Classifiers
Hui Huang Yingqi Qu Jing Liu Muyun Yang Tiejun Zhao
Recently, there has been a growing trend of utilizing Large Language Model
(LLM) to evaluate the quality of other LLMs. Many studies have employed
proprietary close-source models, especially GPT4, as the evaluator.
Alternatively, other works have fine-tuned judge models based on open-source
LLMs as the evaluator. In this study, we conduct an empirical study of
different judge models on their evaluation capability. Our findings indicate
that although the fine-tuned judge models achieve high accuracy on in-domain
test sets, even surpassing GPT4, they are inherently task-specific classifiers,
and their generalizability and fairness severely underperform GPT4.