PARAMANU-AYN: An Efficient Novel Generative and Instruction-tuned
  Language Model for Indian Legal Case Documents
Mitodru Niyogi Arnab Bhattacharya
In this paper, we present PARAMANU-AYN, a language model based exclusively on
case documents of the Supreme Court of India, the Constitution of India, and
the Indian Penal Code. The novel Auto Regressive (AR) decoder based model is
pretrained from scratch at a context size of 8192. We evaluated our pretrained
legal model on perplexity metrics. We also instruction-tuned our pretrained
model on a set of 10,763 instructions covering various legal tasks such as
legal reasoning, judgement explanation, legal clause generation, legal
drafting, legal contract drafting, case summarization, constitutional
question-answering, etc. We also evaluated the responses of prompts for
instruction-tuned models by GPT-3.5-Turbo on clarity, relevance, completeness,
and legal reasoning metrics in a scale of 10. Our model can be run on CPU and
achieved 42.46 tokens/sec CPU inference speed. We found that our models,
despite not being pretrained on legal books, various legal contracts, and legal
documents, were able to learn the domain knowledge required for drafting
various legal contracts and legal clauses, and generalize to draft legal
contracts and legal clauses with limited instruction tuning. Hence, we conclude
that for a strong domain-specialized generative language model (such as legal),
very large amounts of data are not required to develop models from scratch. We
believe that this work is the first attempt to make a dedicated generative
legal language model from scratch for Indian Supreme Court jurisdiction or in
legal NLP overall. We plan to release our Paramanu-Ayn model at
https://www.bharatgpts.com.