Lawyer LLaMA Technical Report
This work addresses the challenge of domain-specific knowledge deficiency in LLMs for legal applications, offering a framework that could benefit legal professionals, though it is incremental as it builds on existing adaptation methods.
The authors tackled the problem of adapting large language models to specialized domains by developing Lawyer LLaMA, a legal domain LLM, which achieved improved performance through domain knowledge injection and expert-written data, outperforming ChatGPT-generated data with hundreds of expert examples surpassing tens of thousands of synthetic ones.
Large Language Models (LLMs), like LLaMA, have exhibited remarkable performance across various tasks. Nevertheless, when deployed to specific domains such as law or medicine, the models still confront the challenge of a deficiency in domain-specific knowledge and an inadequate capability to leverage that knowledge to resolve domain-related problems. In this paper, we propose a new framework to adapt LLMs to specific domains and build Lawyer LLaMA, a legal domain LLM, based on this framework. Specifically, we inject domain knowledge during the continual training stage and teach the model to learn professional skills using properly designed supervised fine-tuning tasks. Moreover, to alleviate the hallucination problem during the model's generation, we add a retrieval module and extract relevant legal articles before the model answers any queries. When learning domain-specific skills, we find that experts' experience is much more useful than experiences distilled from ChatGPT, where hundreds of expert-written data outperform tens of thousands of ChatGPT-generated ones. We will release our model and data.