Chinese Labor Law Large Language Model Benchmark
This work addresses the need for more accurate and reliable AI in legal applications, specifically for Chinese labor law, though it is incremental as it builds on existing methods for domain-specific LLMs.
The authors tackled the problem of large language models struggling with specialized legal subdomains by developing LabourLawLLM, a model tailored to Chinese labor law, and LabourLawBench, a benchmark for diverse tasks, with experiments showing it consistently outperforms general-purpose and existing legal-specific models.
Recent advances in large language models (LLMs) have led to substantial progress in domain-specific applications, particularly within the legal domain. However, general-purpose models such as GPT-4 often struggle with specialized subdomains that require precise legal knowledge, complex reasoning, and contextual sensitivity. To address these limitations, we present LabourLawLLM, a legal large language model tailored to Chinese labor law. We also introduce LabourLawBench, a comprehensive benchmark covering diverse labor-law tasks, including legal provision citation, knowledge-based question answering, case classification, compensation computation, named entity recognition, and legal case analysis. Our evaluation framework combines objective metrics (e.g., ROUGE-L, accuracy, F1, and soft-F1) with subjective assessment based on GPT-4 scoring. Experiments show that LabourLawLLM consistently outperforms general-purpose and existing legal-specific LLMs across task categories. Beyond labor law, our methodology provides a scalable approach for building specialized LLMs in other legal subfields, improving accuracy, reliability, and societal value of legal AI applications.