Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation
  from Text
Nandana Mihindukulasooriya Sanju Tiwari Carlos F. Enguix Kusum Lata
The recent advances in large language models (LLM) and foundation models with
emergent capabilities have been shown to improve the performance of many NLP
tasks. LLMs and Knowledge Graphs (KG) can complement each other such that LLMs
can be used for KG construction or completion while existing KGs can be used
for different tasks such as making LLM outputs explainable or fact-checking in
Neuro-Symbolic manner. In this paper, we present Text2KGBench, a benchmark to
evaluate the capabilities of language models to generate KGs from natural
language text guided by an ontology. Given an input ontology and a set of
sentences, the task is to extract facts from the text while complying with the
given ontology (concepts, relations, domain/range constraints) and being
faithful to the input sentences. We provide two datasets (i) Wikidata-TekGen
with 10 ontologies and 13,474 sentences and (ii) DBpedia-WebNLG with 19
ontologies and 4,860 sentences. We define seven evaluation metrics to measure
fact extraction performance, ontology conformance, and hallucinations by LLMs.
Furthermore, we provide results for two baseline models, Vicuna-13B and
Alpaca-LoRA-13B using automatic prompt generation from test cases. The baseline
results show that there is room for improvement using both Semantic Web and
Natural Language Processing techniques.