LexGPT 0.1: pre-trained GPT-J models with Pile of Law
Jieh-Sheng Lee
This research aims to build generative language models specialized for the
legal domain. The manuscript presents the development of LexGPT models based on
GPT-J models and pre-trained with Pile of Law. The foundation model built in
this manuscript is the initial step for the development of future applications
in the legal domain, such as further training with reinforcement learning from
human feedback. Another objective of this manuscript is to assist legal
professionals in utilizing language models through the ``No Code'' approach. By
fine-tuning models with specialized data and without modifying any source code,
legal professionals can create custom language models for downstream tasks with
minimum effort and technical knowledge. The downstream task in this manuscript
is to turn a LexGPT model into a classifier, although the performance is
notably lower than the state-of-the-art result. How to enhance downstream task
performance without modifying the model or its source code is a research topic
for future exploration.