Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions
Taojun Hu Xiao-Hua Zhou
Natural Language Processing (NLP) is witnessing a remarkable breakthrough
driven by the success of Large Language Models (LLMs). LLMs have gained
significant attention across academia and industry for their versatile
applications in text generation, question answering, and text summarization. As
the landscape of NLP evolves with an increasing number of domain-specific LLMs
employing diverse techniques and trained on various corpus, evaluating
performance of these models becomes paramount. To quantify the performance,
it's crucial to have a comprehensive grasp of existing metrics. Among the
evaluation, metrics which quantifying the performance of LLMs play a pivotal
role. This paper offers a comprehensive exploration of LLM evaluation from a
metrics perspective, providing insights into the selection and interpretation
of metrics currently in use. Our main goal is to elucidate their mathematical
formulations and statistical interpretations. We shed light on the application
of these metrics using recent Biomedical LLMs. Additionally, we offer a
succinct comparison of these metrics, aiding researchers in selecting
appropriate metrics for diverse tasks. The overarching goal is to furnish
researchers with a pragmatic guide for effective LLM evaluation and metric
selection, thereby advancing the understanding and application of these large
language models.