Little Giants: Exploring the Potential of Small LLMs as Evaluation
  Metrics in Summarization in the Eval4NLP 2023 Shared Task
Neema Kotonya Saran Krishnasamy Joel Tetreault Alejandro Jaimes
This paper describes and analyzes our participation in the 2023 Eval4NLP
shared task, which focuses on assessing the effectiveness of prompt-based
techniques to empower Large Language Models to handle the task of quality
estimation, particularly in the context of evaluating machine translations and
summaries. We conducted systematic experiments with various prompting
techniques, including standard prompting, prompts informed by annotator
instructions, and innovative chain-of-thought prompting. In addition, we
integrated these approaches with zero-shot and one-shot learning methods to
maximize the efficacy of our evaluation procedures. Our work reveals that
combining these approaches using a "small", open source model (orca_mini_v3_7B)
yields competitive results.