ALARB: An Arabic Legal Argument Reasoning Benchmark
This addresses the need for better evaluation tools for Arabic LLMs in legal reasoning, though it is incremental as it adapts existing benchmarking approaches to a specific domain and language.
The authors tackled the lack of datasets for evaluating multistep reasoning in Arabic large language models (LLMs) by introducing ALARB, a benchmark based on over 13K Saudi commercial court cases, and showed that instruction-tuning a 12B parameter model with it significantly improved performance in verdict prediction and generation to levels comparable to GPT-4o.
We introduce ALARB, a dataset and suite of tasks designed to evaluate the reasoning capabilities of large language models (LLMs) within the Arabic legal domain. While existing Arabic benchmarks cover some knowledge-intensive tasks such as retrieval and understanding, substantial datasets focusing specifically on multistep reasoning for Arabic LLMs, especially in open-ended contexts, are lacking. The dataset comprises over 13K commercial court cases from Saudi Arabia, with each case including the facts presented, the reasoning of the court, the verdict, as well as the cited clauses extracted from the regulatory documents. We define a set of challenging tasks leveraging this dataset and reflecting the complexity of real-world legal reasoning, including verdict prediction, completion of reasoning chains in multistep legal arguments, and identification of relevant regulations based on case facts. We benchmark a representative selection of current open and closed Arabic LLMs on these tasks and demonstrate the dataset's utility for instruction tuning. Notably, we show that instruction-tuning a modest 12B parameter model using ALARB significantly enhances its performance in verdict prediction and Arabic verdict generation, reaching a level comparable to that of GPT-4o.