Evaluating Factual Consistency of Texts with Semantic Role Labeling
Jing Fan Dennis Aumiller Michael Gertz
Automated evaluation of text generation systems has recently seen increasing
attention, particularly checking whether generated text stays truthful to input
sources. Existing methods frequently rely on an evaluation using task-specific
language models, which in turn allows for little interpretability of generated
scores. We introduce SRLScore, a reference-free evaluation metric designed with
text summarization in mind. Our approach generates fact tuples constructed from
Semantic Role Labels, applied to both input and summary texts. A final
factuality score is computed by an adjustable scoring mechanism, which allows
for easy adaption of the method across domains. Correlation with human
judgments on English summarization datasets shows that SRLScore is competitive
with state-of-the-art methods and exhibits stable generalization across
datasets without requiring further training or hyperparameter tuning. We
experiment with an optional co-reference resolution step, but find that the
performance boost is mostly outweighed by the additional compute required. Our
metric is available online at https://github.com/heyjing/SRLScore.