Unlocking Practical Applications in Legal Domain: Evaluation of GPT for
  Zero-Shot Semantic Annotation of Legal Texts
Jaromir Savelka
We evaluated the capability of a state-of-the-art generative pre-trained
transformer (GPT) model to perform semantic annotation of short text snippets
(one to few sentences) coming from legal documents of various types.
Discussions of potential uses (e.g., document drafting, summarization) of this
emerging technology in legal domain have intensified, but to date there has not
been a rigorous analysis of these large language models' (LLM) capacity in
sentence-level semantic annotation of legal texts in zero-shot learning
settings. Yet, this particular type of use could unlock many practical
applications (e.g., in contract review) and research opportunities (e.g., in
empirical legal studies). We fill the gap with this study. We examined if and
how successfully the model can semantically annotate small batches of short
text snippets (10-50) based exclusively on concise definitions of the semantic
types. We found that the GPT model performs surprisingly well in zero-shot
settings on diverse types of documents (F1=.73 on a task involving court
opinions, .86 for contracts, and .54 for statutes and regulations). These
findings can be leveraged by legal scholars and practicing lawyers alike to
guide their decisions in integrating LLMs in wide range of workflows involving
semantic annotation of legal texts.