AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios
This addresses the problem of analyzing appellate scenarios for LegalAI researchers, providing a new benchmark but is incremental as it extends existing datasets to a specific domain.
The authors tackled the lack of focus on appellate processes in LegalAI by introducing the AppealCase dataset, which includes 10,000 matched civil case documents with detailed annotations, and found that current models achieve less than 50% F1 scores on judgment reversal prediction.
Recent advances in LegalAI have primarily focused on individual case judgment analysis, often overlooking the critical appellate process within the judicial system. Appeals serve as a core mechanism for error correction and ensuring fair trials, making them highly significant both in practice and in research. To address this gap, we present the AppealCase dataset, consisting of 10,000 pairs of real-world, matched first-instance and second-instance documents across 91 categories of civil cases. The dataset also includes detailed annotations along five dimensions central to appellate review: judgment reversals, reversal reasons, cited legal provisions, claim-level decisions, and whether there is new information in the second instance. Based on these annotations, we propose five novel LegalAI tasks and conduct a comprehensive evaluation across 20 mainstream models. Experimental results reveal that all current models achieve less than 50% F1 scores on the judgment reversal prediction task, highlighting the complexity and challenge of the appeal scenario. We hope that the AppealCase dataset will spur further research in LegalAI for appellate case analysis and contribute to improving consistency in judicial decision-making.