Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation
Jiatong Li Rui Li Qi Liu
Large Language Models (LLMs) have made progress in various real-world tasks,
which stimulates requirements for the evaluation of LLMs. Existing LLM
evaluation methods are mainly supervised signal-based which depends on static
datasets and cannot evaluate the ability of LLMs in dynamic real-world
scenarios where deep interaction widely exists. Other LLM evaluation methods
are human-based which are costly and time-consuming and are incapable of
large-scale evaluation of LLMs. To address the issues above, we propose a novel
Deep Interaction-based LLM-evaluation framework. In our proposed framework,
LLMs' performances in real-world domains can be evaluated from their deep
interaction with other LLMs in elaborately designed evaluation tasks.
Furthermore, our proposed framework is a general evaluation method that can be
applied to a host of real-world tasks such as machine translation and code
generation. We demonstrate the effectiveness of our proposed method through
extensive experiments on four elaborately designed evaluation tasks.