Datasets for Portuguese Legal Semantic Textual Similarity: Comparing weak supervision and an annotation process approaches
This work addresses the problem of dataset scarcity for AI applications in the Brazilian legal system, but it is incremental as it focuses on creating and evaluating datasets rather than developing new methods.
The authors tackled the scarcity of labeled datasets in the legal domain by contributing four datasets for Portuguese legal semantic textual similarity, including two labeled using a heuristic, and found that the heuristic labels are useful compared to a small expert-annotated ground truth.
The Brazilian judiciary has a large workload, resulting in a long time to finish legal proceedings. Brazilian National Council of Justice has established in Resolution 469/2022 formal guidance for document and process digitalization opening up the possibility of using automatic techniques to help with everyday tasks in the legal field, particularly in a large number of texts yielded on the routine of law procedures. Notably, Artificial Intelligence (AI) techniques allow for processing and extracting useful information from textual data, potentially speeding up the process. However, datasets from the legal domain required by several AI techniques are scarce and difficult to obtain as they need labels from experts. To address this challenge, this article contributes with four datasets from the legal domain, two with documents and metadata but unlabeled, and another two labeled with a heuristic aiming at its use in textual semantic similarity tasks. Also, to evaluate the effectiveness of the proposed heuristic label process, this article presents a small ground truth dataset generated from domain expert annotations. The analysis of ground truth labels highlights that semantic analysis of domain text can be challenging even for domain experts. Also, the comparison between ground truth and heuristic labels shows that heuristic labels are useful.