LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset
This addresses a gap in legal research by enabling more accurate dataset extraction for computational law studies, though it is incremental as it applies existing LLM technology to a specific domain task.
The study tackled the problem of efficiently identifying summary judgment cases from a large corpus of UK court decisions, comparing a keyword-based NLP method with a Claude 2 LLM approach, where the LLM achieved a weighted F1 score of 0.94 versus 0.78 for keywords and extracted 3,102 cases.
To undertake computational research of the law, efficiently identifying datasets of court decisions that relate to a specific legal issue is a crucial yet challenging endeavour. This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We introduce a comparative analysis of two computational methods: (1) a traditional natural language processing-based approach leveraging expert-generated keywords and logical operators and (2) an innovative application of the Claude 2 large language model to classify cases based on content-specific prompts. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. Despite iterative refinement, the search logic based on keywords fails to capture nuances in legal language. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span. The paper marks a pioneering step in employing advanced natural language processing to tackle core legal research tasks, demonstrating how these technologies can bridge systemic gaps and enhance the accessibility of legal information. We share the extracted dataset metrics to support further research on summary judgments.