DecompEval: Evaluating Generated Texts as Unsupervised Decomposed
  Question Answering
Pei Ke Fei Huang Fei Mi Yasheng Wang Qun Liu Xiaoyan Zhu Minlie Huang
Existing evaluation metrics for natural language generation (NLG) tasks face
the challenges on generalization ability and interpretability. Specifically,
most of the well-performed metrics are required to train on evaluation datasets
of specific NLG tasks and evaluation dimensions, which may cause over-fitting
to task-specific datasets. Furthermore, existing metrics only provide an
evaluation score for each dimension without revealing the evidence to interpret
how this score is obtained. To deal with these challenges, we propose a simple
yet effective metric called DecompEval. This metric formulates NLG evaluation
as an instruction-style question answering task and utilizes instruction-tuned
pre-trained language models (PLMs) without training on evaluation datasets,
aiming to enhance the generalization ability. To make the evaluation process
more interpretable, we decompose our devised instruction-style question about
the quality of generated texts into the subquestions that measure the quality
of each sentence. The subquestions with their answers generated by PLMs are
then recomposed as evidence to obtain the evaluation result. Experimental
results show that DecompEval achieves state-of-the-art performance in untrained
metrics for evaluating text summarization and dialogue generation, which also
exhibits strong dimension-level / task-level generalization ability and
interpretability.