FEEL: A Framework for Evaluating Emotional Support Capability with Large
  Language Models
Huaiwen Zhang Yu Chen Ming Wang Shi Feng
Emotional Support Conversation (ESC) is a typical dialogue that can
effec-tively assist the user in mitigating emotional pressures. However, owing
to the inherent subjectivity involved in analyzing emotions, current
non-artificial methodologies face challenges in effectively appraising the
emo-tional support capability. These metrics exhibit a low correlation with
human judgments. Concurrently, manual evaluation methods extremely will cause
high costs. To solve these problems, we propose a novel model FEEL (Framework
for Evaluating Emotional Support Capability with Large Lan-guage Models),
employing Large Language Models (LLMs) as evaluators to assess emotional
support capabilities. The model meticulously considers var-ious evaluative
aspects of ESC to apply a more comprehensive and accurate evaluation method for
ESC. Additionally, it employs a probability distribu-tion approach for a more
stable result and integrates an ensemble learning strategy, leveraging multiple
LLMs with assigned weights to enhance evalua-tion accuracy. To appraise the
performance of FEEL, we conduct extensive experiments on existing ESC model
dialogues. Experimental results demon-strate our model exhibits a substantial
enhancement in alignment with human evaluations compared to the baselines. Our
source code is available at https://github.com/Ansisy/FEEL.