Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities
This provides a new benchmark for evaluating legal reasoning in LLMs, addressing a domain-specific need in AI and legal technology, though it is incremental in benchmarking methodology.
The authors tackled the problem of evaluating language models' legal reasoning capabilities independently of domain-specific knowledge by introducing the Korean Canonical Legal Benchmark (KCL), which includes multiple-choice and open-ended questions with supporting precedents. Their evaluation of over 30 models revealed significant performance gaps, especially in open-ended tasks, and showed that reasoning-specialized models consistently outperform general-purpose ones.
We introduce the Korean Canonical Legal Benchmark (KCL), a benchmark designed to assess language models' legal reasoning capabilities independently of domain-specific knowledge. KCL provides question-level supporting precedents, enabling a more faithful disentanglement of reasoning ability from parameterized knowledge. KCL consists of two components: (1) KCL-MCQA, multiple-choice problems of 283 questions with 1,103 aligned precedents, and (2) KCL-Essay, open-ended generation problems of 169 questions with 550 aligned precedents and 2,739 instance-level rubrics for automated evaluation. Our systematic evaluation of 30+ models shows large remaining gaps, particularly in KCL-Essay, and that reasoning-specialized models consistently outperform their general-purpose counterparts. We release all resources, including the benchmark dataset and evaluation code, at https://github.com/lbox-kr/kcl.