Which is better? Exploring Prompting Strategy For LLM-based Metrics
Joonghoon Kim Saeran Park Kiyoon Jeong Sangmin Lee Seung Hun Han Jiyoon Lee Pilsung Kang
This paper describes the DSBA submissions to the Prompting Large Language
Models as Explainable Metrics shared task, where systems were submitted to two
tracks: small and large summarization tracks. With advanced Large Language
Models (LLMs) such as GPT-4, evaluating the quality of Natural Language
Generation (NLG) has become increasingly paramount. Traditional
similarity-based metrics such as BLEU and ROUGE have shown to misalign with
human evaluation and are ill-suited for open-ended generation tasks. To address
this issue, we explore the potential capability of LLM-based metrics,
especially leveraging open-source LLMs. In this study, wide range of prompts
and prompting techniques are systematically analyzed with three approaches:
prompting strategy, score aggregation, and explainability. Our research focuses
on formulating effective prompt templates, determining the granularity of NLG
quality scores and assessing the impact of in-context examples on LLM-based
evaluation. Furthermore, three aggregation strategies are compared to identify
the most reliable method for aggregating NLG quality scores. To examine
explainability, we devise a strategy that generates rationales for the scores
and analyzes the characteristics of the explanation produced by the open-source
LLMs. Extensive experiments provide insights regarding evaluation capabilities
of open-source LLMs and suggest effective prompting strategies.