The Massive Legal Embedding Benchmark (MLEB)
This provides a standardized evaluation tool for researchers and practitioners in legal AI, though it is incremental as it builds on existing benchmarks by expanding scope and diversity.
The authors tackled the lack of a comprehensive benchmark for legal information retrieval by creating the Massive Legal Embedding Benchmark (MLEB), which includes ten expert-annotated datasets across multiple jurisdictions and document types, with seven newly constructed datasets to fill gaps.
We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.