Using Large Language Models to Support Thematic Analysis in Empirical
  Legal Studies
Jakub Drápal Hannes Westermann Jaromir Savelka
Thematic analysis and other variants of inductive coding are widely used
qualitative analytic methods within empirical legal studies (ELS). We propose a
novel framework facilitating effective collaboration of a legal expert with a
large language model (LLM) for generating initial codes (phase 2 of thematic
analysis), searching for themes (phase 3), and classifying the data in terms of
the themes (to kick-start phase 4). We employed the framework for an analysis
of a dataset (n=785) of facts descriptions from criminal court opinions
regarding thefts. The goal of the analysis was to discover classes of typical
thefts. Our results show that the LLM, namely OpenAI's GPT-4, generated
reasonable initial codes, and it was capable of improving the quality of the
codes based on expert feedback. They also suggest that the model performed well
in zero-shot classification of facts descriptions in terms of the themes.
Finally, the themes autonomously discovered by the LLM appear to map fairly
well to the themes arrived at by legal experts. These findings can be leveraged
by legal researchers to guide their decisions in integrating LLMs into their
thematic analyses, as well as other inductive coding projects.