How Reliable Are Automatic Evaluation Methods for Instruction-Tuned
  LLMs?
Ehsan Doostmohammadi Oskar Holmström Marco Kuhlmann
Work on instruction-tuned Large Language Models (LLMs) has used automatic
methods based on text overlap and LLM judgments as cost-effective alternatives
to human evaluation. In this paper, we study the reliability of such methods
across a broad range of tasks and in a cross-lingual setting. In contrast to
previous findings, we observe considerable variability in correlations between
automatic methods and human evaluators when scores are differentiated by task
type. Specifically, the widely-used ROUGE-L metric strongly correlates with
human judgments for short-answer English tasks but is unreliable in free-form
generation tasks and cross-lingual transfer. The effectiveness of GPT-4 as an
evaluator depends on including reference answers when prompting for
assessments, which can lead to overly strict evaluations in free-form
generation tasks. In summary, we find that, while automatic evaluation methods
can approximate human judgements under specific conditions, their reliability
is highly context-dependent. Our findings enhance the understanding of how
automatic methods should be applied and interpreted when developing and
evaluating instruction-tuned LLMs.