Evaluating Generative Models for Graph-to-Text Generation
Shuzhou Yuan Michael Färber
Large language models (LLMs) have been widely employed for graph-to-text
generation tasks. However, the process of finetuning LLMs requires significant
training resources and annotation work. In this paper, we explore the
capability of generative models to generate descriptive text from graph data in
a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two
graph-to-text datasets and compare their performance with that of finetuned LLM
models such as T5 and BART. Our results demonstrate that generative models are
capable of generating fluent and coherent text, achieving BLEU scores of 10.57
and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error
analysis reveals that generative models still struggle with understanding the
semantic relations between entities, and they also tend to generate text with
hallucinations or irrelevant information. As a part of error analysis, we
utilize BERT to detect machine-generated text and achieve high macro-F1 scores.
We have made the text generated by generative models publicly available.