Automatic Evaluation of Generative Models with Instruction Tuning
Shuhaib Mehri Vered Shwartz
Automatic evaluation of natural language generation has long been an elusive
goal in NLP.A recent paradigm fine-tunes pre-trained language models to emulate
human judgements for a particular task and evaluation criterion. Inspired by
the generalization ability of instruction-tuned models, we propose a learned
metric based on instruction tuning. To test our approach, we collected HEAP, a
dataset of human judgements across various NLG tasks and evaluation criteria.
Our findings demonstrate that instruction tuning language models on HEAP yields
good performance on many evaluation tasks, though some criteria are less
trivial to learn than others. Further, jointly training on multiple tasks can
yield additional performance improvements, which can be beneficial for future
tasks with little to no human annotated data.