FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation
  of Large Language Models
Zhuohao Yu Chang Gao Wenjin Yao Yidong Wang Zhengran Zeng Wei Ye Jindong Wang Yue Zhang Shikun Zhang
The rapid development of large language model (LLM) evaluation methodologies
and datasets has led to a profound challenge: integrating state-of-the-art
evaluation techniques cost-effectively while ensuring reliability,
reproducibility, and efficiency. Currently, there is a notable absence of a
unified and adaptable framework that seamlessly integrates various evaluation
approaches. Moreover, the reliability of evaluation findings is often
questionable due to potential data contamination, with the evaluation
efficiency commonly overlooked when facing the substantial costs associated
with LLM inference. In response to these challenges, we introduce FreeEval, a
modular and scalable framework crafted to enable trustworthy and efficient
automatic evaluations of LLMs. Firstly, FreeEval's unified abstractions
simplify the integration and improve the transparency of diverse evaluation
methodologies, encompassing dynamic evaluation that demand sophisticated LLM
interactions. Secondly, the framework integrates meta-evaluation techniques
like human evaluation and data contamination detection, which, along with
dynamic evaluation modules in the platform, enhance the fairness of the
evaluation outcomes. Lastly, FreeEval is designed with a high-performance
infrastructure, including distributed computation and caching strategies,
enabling extensive evaluations across multi-node, multi-GPU clusters for
open-source and proprietary LLMs.