LexGPT 0.1: pre-trained GPT-J models with Pile of Law
This work addresses the need for accessible AI tools in the legal domain, but it is incremental as it builds on existing models without major methodological innovations.
The researchers developed LexGPT, a generative language model specialized for the legal domain by pre-training GPT-J models with Pile of Law, aiming to assist legal professionals through a 'No Code' approach for fine-tuning, though downstream classification performance was notably lower than state-of-the-art.
This research aims to build generative language models specialized for the legal domain. The manuscript presents the development of LexGPT models based on GPT-J models and pre-trained with Pile of Law. The foundation model built in this manuscript is the initial step for the development of future applications in the legal domain, such as further training with reinforcement learning from human feedback. Another objective of this manuscript is to assist legal professionals in utilizing language models through the ``No Code'' approach. By fine-tuning models with specialized data and without modifying any source code, legal professionals can create custom language models for downstream tasks with minimum effort and technical knowledge. The downstream task in this manuscript is to turn a LexGPT model into a classifier, although the performance is notably lower than the state-of-the-art result. How to enhance downstream task performance without modifying the model or its source code is a research topic for future exploration.