GPT Struct Me: Probing GPT Models on Narrative Entity Extraction
Hugo Sousa Nuno Guimarães Alípio Jorge Ricardo Campos
The importance of systems that can extract structured information from
textual data becomes increasingly pronounced given the ever-increasing volume
of text produced on a daily basis. Having a system that can effectively extract
such information in an interoperable manner would be an asset for several
domains, be it finance, health, or legal. Recent developments in natural
language processing led to the production of powerful language models that can,
to some degree, mimic human intelligence. Such effectiveness raises a pertinent
question: Can these models be leveraged for the extraction of structured
information? In this work, we address this question by evaluating the
capabilities of two state-of-the-art language models -- GPT-3 and GPT-3.5,
commonly known as ChatGPT -- in the extraction of narrative entities, namely
events, participants, and temporal expressions. This study is conducted on the
Text2Story Lusa dataset, a collection of 119 Portuguese news articles whose
annotation framework includes a set of entity structures along with several
tags and attribute values. We first select the best prompt template through an
ablation study over prompt components that provide varying degrees of
information on a subset of documents of the dataset. Subsequently, we use the
best templates to evaluate the effectiveness of the models on the remaining
documents. The results obtained indicate that GPT models are competitive with
out-of-the-box baseline systems, presenting an all-in-one alternative for
practitioners with limited resources. By studying the strengths and limitations
of these models in the context of information extraction, we offer insights
that can guide future improvements and avenues to explore in this field.