ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US
This provides a more realistic benchmark for legal NLP in common law systems, though it is incremental as it adapts existing methods to a new dataset.
The authors tackled the lack of a challenging Legal Judgment Prediction dataset for US class action cases by releasing a new dataset based on complaints rather than court summaries, and their Longformer model achieved 63% accuracy, outperforming human experts at 53%.
The research field of Legal Natural Language Processing (NLP) has been very active recently, with Legal Judgment Prediction (LJP) becoming one of the most extensively studied tasks. To date, most publicly released LJP datasets originate from countries with civil law. In this work, we release, for the first time, a challenging LJP dataset focused on class action cases in the US. It is the first dataset in the common law system that focuses on the harder and more realistic task involving the complaints as input instead of the often used facts summary written by the court. Additionally, we study the difficulty of the task by collecting expert human predictions, showing that even human experts can only reach 53% accuracy on this dataset. Our Longformer model clearly outperforms the human baseline (63%), despite only considering the first 2,048 tokens. Furthermore, we perform a detailed error analysis and find that the Longformer model is significantly better calibrated than the human experts. Finally, we publicly release the dataset and the code used for the experiments.