Legal Retrieval for Public Defenders
For public defenders overwhelmed by caseloads, this work provides a domain-specific retrieval tool and benchmark, though the improvement is incremental and domain-limited.
The paper develops the NJ BriefBank, a retrieval tool for public defenders, showing that existing benchmarks fail in this domain but adding domain knowledge (query expansion, curated synthetic examples) improves retrieval quality. They release a taxonomy of defender queries and a manually annotated evaluation dataset.
AI tools are suggested as solutions to assist public agencies with heavy workloads. In public defense -- where a constitutional right to counsel meets the complexities of law, overwhelming caseloads, and constrained resources -- practitioners face especially taxing conditions. Yet, there is little evidence of how AI could meaningfully support defenders' day-to-day work. In partnership with the New Jersey Office of the Public Defender, we develop the NJ BriefBank, a retrieval tool which surfaces relevant appellate briefs to streamline legal research and writing. We show that existing retrieval benchmarks fail to transfer to real public defense research, however adding domain knowledge improves retrieval quality. This includes query expansion with legal reasoning, domain-specific data and curated synthetic examples. To facilitate further research, we release a taxonomy of realistic defender search queries and a manually annotated evaluation dataset for public defense retrieval. This benchmark is highly correlated with a proprietary retrieval dataset annotated by experienced public defenders. Our work improves on the status quo of realistic legal retrieval benchmarking and illustrates one approach to applying AI in a real-world public interest setting.