AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction
This addresses the need for efficient appellate review in legal practice due to case volume surges, but it is incremental as it adapts anomaly detection to a new domain-specific task.
The paper tackles the problem of errors in legal judgments by introducing the APPELLATE REVIEW task for detecting, classifying, and correcting errors, and constructs the AR-BENCH dataset with 8,700 annotated decisions and 34,617 supplementary corpora, revealing limitations in 14 large language models' ability to identify legal application errors.
Legal judgments may contain errors due to the complexity of case circumstances and the abstract nature of legal concepts, while existing appellate review mechanisms face efficiency pressures from a surge in case volumes. Although current legal AI research focuses on tasks like judgment prediction and legal document generation, the task of judgment review differs fundamentally in its objectives and paradigm: it centers on detecting, classifying, and correcting errors after a judgment is issued, constituting anomaly detection rather than prediction or generation. To address this research gap, we introduce a novel task APPELLATE REVIEW, aiming to assess models' diagnostic reasoning and reliability in legal practice. We also construct a novel dataset benchmark AR-BENCH, which comprises 8,700 finely annotated decisions and 34,617 supplementary corpora. By evaluating 14 large language models, we reveal critical limitations in existing models' ability to identify legal application errors, providing empirical evidence for future improvements.