Benchmarking the Abilities of Large Language Models for RDF Knowledge
  Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?
Johannes Frey Lars-Peter Meyer Natanael Arndt Felix Brei Kirill Bulert
Large Language Models (LLMs) are advancing at a rapid pace, with significant
improvements at natural language processing and coding tasks. Yet, their
ability to work with formal languages representing data, specifically within
the realm of knowledge graph engineering, remains under-investigated. To
evaluate the proficiency of various LLMs, we created a set of five tasks that
probe their ability to parse, understand, analyze, and create knowledge graphs
serialized in Turtle syntax. These tasks, each embodying distinct degrees of
complexity and being able to scale with the size of the problem, have been
integrated into our automated evaluation system, the LLM-KG-Bench. The
evaluation encompassed four commercially available LLMs - GPT-3.5, GPT-4,
Claude 1.3, and Claude 2.0, as well as two freely accessible offline models,
GPT4All Vicuna and GPT4All Falcon 13B. This analysis offers an in-depth
understanding of the strengths and shortcomings of LLMs in relation to their
application within RDF knowledge graph engineering workflows utilizing Turtle
representation. While our findings show that the latest commercial models
outperform their forerunners in terms of proficiency with the Turtle language,
they also reveal an apparent weakness. These models fall short when it comes to
adhering strictly to the output formatting constraints, a crucial requirement
in this context.