LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding
This work addresses the problem of evaluating legal-specific LLMs for contract understanding, providing a holistic benchmark for researchers and developers in legal NLP, though it is incremental as it focuses on evaluation rather than new method development.
The paper tackled the lack of comprehensive evaluation for legal-specific LLMs in contract understanding by testing 10 such models against 7 general-purpose LLMs on three tasks, finding that legal-specific models consistently outperformed general ones, with Legal-BERT and Contracts-BERT achieving new SOTAs on two tasks despite having 69% fewer parameters.
Despite advances in legal NLP, no comprehensive evaluation covering multiple legal-specific LLMs currently exists for contract classification tasks in contract understanding. To address this gap, we present an evaluation of 10 legal-specific LLMs on three English language contract understanding tasks and compare them with 7 general-purpose LLMs. The results show that legal-specific LLMs consistently outperform general-purpose models, especially on tasks requiring nuanced legal understanding. Legal-BERT and Contracts-BERT establish new SOTAs on two of the three tasks, despite having 69% fewer parameters than the best-performing general-purpose LLM. We also identify CaseLaw-BERT and LexLM as strong additional baselines for contract understanding. Our results provide a holistic evaluation of legal-specific LLMs and will facilitate the development of more accurate contract understanding systems.