Self-Evaluation of Large Language Model based on Glass-box Features
Hui Huang Yingqi Qu Jing Liu Muyun Yang Tiejun Zhao
The proliferation of open-source Large Language Models (LLMs) underscores the
pressing need for evaluation methods. Existing works primarily rely on external
evaluators, focusing on training and prompting strategies. However, a crucial
aspect - model-aware glass-box features - is overlooked. In this study, we
explore the utility of glass-box features under the scenario of
self-evaluation, namely applying an LLM to evaluate its own output. We
investigate various glass-box feature groups and discovered that the softmax
distribution serves as a reliable indicator for quality evaluation.
Furthermore, we propose two strategies to enhance the evaluation by
incorporating features derived from references. Experimental results on public
benchmarks validate the feasibility of self-evaluation of LLMs using glass-box
features.