LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK
  Case Law Dataset
Ahmed Izzidien Holli Sargeant Felix Steffek
To undertake computational research of the law, efficiently identifying
datasets of court decisions that relate to a specific legal issue is a crucial
yet challenging endeavour. This study addresses the gap in the literature
working with large legal corpora about how to isolate cases, in our case
summary judgments, from a large corpus of UK court decisions. We introduce a
comparative analysis of two computational methods: (1) a traditional natural
language processing-based approach leveraging expert-generated keywords and
logical operators and (2) an innovative application of the Claude 2 large
language model to classify cases based on content-specific prompts. We use the
Cambridge Law Corpus of 356,011 UK court decisions and determine that the large
language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords.
Despite iterative refinement, the search logic based on keywords fails to
capture nuances in legal language. We identify and extract 3,102 summary
judgment cases, enabling us to map their distribution across various UK courts
over a temporal span. The paper marks a pioneering step in employing advanced
natural language processing to tackle core legal research tasks, demonstrating
how these technologies can bridge systemic gaps and enhance the accessibility
of legal information. We share the extracted dataset metrics to support further
research on summary judgments.