LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases
This work addresses a gap in legal AI for Chinese civil law by providing a benchmark, but it is incremental as it builds on existing relation extraction methods.
The authors tackled the problem of legal relation extraction in Chinese civil cases by introducing a comprehensive schema and benchmark called LexRel, and found that current large language models have significant limitations in accurately identifying these relations, with performance gains on downstream tasks when incorporating legal relations information.
Legal relations form a highly consequential analytical framework of civil law system, serving as a crucial foundation for resolving disputes and realizing values of the rule of law in judicial practice. However, legal relations in Chinese civil cases remain underexplored in the field of legal artificial intelligence (legal AI), largely due to the absence of comprehensive schemas. In this work, we firstly introduce a comprehensive schema, which contains a hierarchical taxonomy and definitions of arguments, for AI systems to capture legal relations in Chinese civil cases. Based on this schema, we then formulate legal relation extraction task and present LexRel, an expert-annotated benchmark for legal relation extraction in Chinese civil law. We use LexRel to evaluate state-of-the-art large language models (LLMs) on legal relation extractions, showing that current LLMs exhibit significant limitations in accurately identifying civil legal relations. Furthermore, we demonstrate that incorporating legal relations information leads to consistent performance gains on other downstream legal AI tasks.