From Text to Structure: Using Large Language Models to Support the
  Development of Legal Expert Systems
Samyar Janatian Hannes Westermann Jinzhe Tan Jaromir Savelka Karim Benyekhlef
Encoding legislative text in a formal representation is an important
prerequisite to different tasks in the field of AI & Law. For example,
rule-based expert systems focused on legislation can support laypeople in
understanding how legislation applies to them and provide them with helpful
context and information. However, the process of analyzing legislation and
other sources to encode it in the desired formal representation can be
time-consuming and represents a bottleneck in the development of such systems.
Here, we investigate to what degree large language models (LLMs), such as
GPT-4, are able to automatically extract structured representations from
legislation. We use LLMs to create pathways from legislation, according to the
JusticeBot methodology for legal decision support systems, evaluate the
pathways and compare them to manually created pathways. The results are
promising, with 60% of generated pathways being rated as equivalent or better
than manually created ones in a blind comparison. The approach suggests a
promising path to leverage the capabilities of LLMs to ease the costly
development of systems based on symbolic approaches that are transparent and
explainable.