Exploring Large Language Models and Hierarchical Frameworks for
  Classification of Large Unstructured Legal Documents
Nishchal Prasad Mohand Boughanem Taoufiq Dkaki
Legal judgment prediction suffers from the problem of long case documents
exceeding tens of thousands of words, in general, and having a non-uniform
structure. Predicting judgments from such documents becomes a challenging task,
more so on documents with no structural annotation. We explore the
classification of these large legal documents and their lack of structural
information with a deep-learning-based hierarchical framework which we call
MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment
prediction. Specifically, we divide a document into parts to extract their
embeddings from the last four layers of a custom fine-tuned Large Language
Model, and try to approximate their structure through unsupervised clustering.
Which we use in another set of transformer encoder layers to learn the
inter-chunk representations. We analyze the adaptability of Large Language
Models (LLMs) with multi-billion parameters (GPT-Neo, and GPT-J) with the
hierarchical framework of MESc and compare them with their standalone
performance on legal texts. We also study their intra-domain(legal) transfer
learning capability and the impact of combining embeddings from their last
layers in MESc. We test these methods and their effectiveness with extensive
experiments and ablation studies on legal documents from India, the European
Union, and the United States with the ILDC dataset and a subset of the LexGLUE
dataset. Our approach achieves a minimum total performance gain of
approximately 2 points over previous state-of-the-art methods.