PANDA (Pedantic ANswer-correctness Determination and
  Adjudication):Improving Automatic Evaluation for Question Answering and Text
  Generation
Zongxia Li Ishani Mondal Yijun Liang Huy Nghiem Jordan Lee Boyd-Graber
Question answering (QA) can only make progress if we know if an answer is
correct, but for many of the most challenging and interesting QA examples,
current answer correctness (AC) metrics do not align with human judgments,
particularly verbose, free form answers from large language models (LLM). There
are two challenges: a lack of data and that models are too big. LLM based
scorers correlate better with humans, but this expensive task has only been
tested on limited QA datasets. We rectify these issues by providing clear
guidelines for evaluating machine QA adopted from human QA contests. We also
introduce Precise ANswer correctness Determination and Adjudication (PANDA), a
small, efficient, deterministic AC classifier (812 KB) that more accurately
evaluates answer correctness.