Passing the Brazilian OAB Exam: data preparation and some experiments
This work addresses the challenge of benchmarking legal information systems for legal professionals, but it is incremental as it focuses on data preparation and initial experiments.
The authors tackled the problem of legal reasoning by creating a dataset from the Brazilian OAB bar exam and conducting preliminary experiments to find justifications for answers, resulting in baseline performance measures for future evaluation.
In Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, the national bar exams. The OAB exams therefore provide an excellent benchmark for the performance of legal information systems since passing the exam would arguably signal that the system has acquired capacity of legal reasoning comparable to that of a human lawyer. This article describes the construction of a new data set and some preliminary experiments on it, treating the problem of finding the justification for the answers to questions. The results provide a baseline performance measure against which to evaluate future improvements. We discuss the reasons to the poor performance and propose next steps.