Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models
This addresses the challenge for lawyers and legal professionals in automating complex procedural tasks, but it is incremental as it highlights limitations rather than breakthroughs.
The study tackled the problem of automating legal citation formatting using large language models (LLMs) by testing them on a dataset of 866 Bluebook tasks, finding that models produced fully compliant citations only 69%-74% of the time, with in-context learning improving accuracy to just 77%.
Legal practice requires careful adherence to procedural rules. In the United States, few are more complex than those found in The Bluebook: A Uniform System of Citation. Compliance with this system's 500+ pages of byzantine formatting instructions is the raison d'etre of thousands of student law review editors and the bete noire of lawyers everywhere. To evaluate whether large language models (LLMs) are able to adhere to the procedures of such a complicated system, we construct an original dataset of 866 Bluebook tasks and test flagship LLMs from OpenAI, Anthropic, Google, Meta, and DeepSeek. We show (1) that these models produce fully compliant Bluebook citations only 69%-74% of the time and (2) that in-context learning on the Bluebook's underlying system of rules raises accuracy only to 77%. These results caution against using off-the-shelf LLMs to automate aspects of the law where fidelity to procedure is paramount.