Prediction of Arabic Legal Rulings using Large Language Models
Adel Ammar Anis Koubaa Bilel Benjdira Omar Najar Serry Sibaee
In the intricate field of legal studies, the analysis of court decisions is a
cornerstone for the effective functioning of the judicial system. The ability
to predict court outcomes helps judges during the decision-making process and
equips lawyers with invaluable insights, enhancing their strategic approaches
to cases. Despite its significance, the domain of Arabic court analysis remains
under-explored. This paper pioneers a comprehensive predictive analysis of
Arabic court decisions on a dataset of 10,813 commercial court real cases,
leveraging the advanced capabilities of the current state-of-the-art large
language models. Through a systematic exploration, we evaluate three prevalent
foundational models (LLaMA-7b, JAIS-13b, and GPT3.5-turbo) and three training
paradigms: zero-shot, one-shot, and tailored fine-tuning. Besides, we assess
the benefit of summarizing and/or translating the original Arabic input texts.
This leads to a spectrum of 14 model variants, for which we offer a granular
performance assessment with a series of different metrics (human assessment,
GPT evaluation, ROUGE, and BLEU scores). We show that all variants of LLaMA
models yield limited performance, whereas GPT-3.5-based models outperform all
other models by a wide margin, surpassing the average score of the dedicated
Arabic-centric JAIS model by 50%. Furthermore, we show that all scores except
human evaluation are inconsistent and unreliable for assessing the performance
of large language models on court decision predictions. This study paves the
way for future research, bridging the gap between computational linguistics and
Arabic legal analytics.