LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation
For legal NLP researchers, it provides a structured evaluation method for legal proposition generation, but the findings are incremental.
The paper introduces LP-Eval, a rubric and dataset for evaluating legal proposition generation by LLMs, finding that LLMs produce high-quality propositions but that rubric-guided LLM judgments align better with experts than direct scoring, though they miss fine-grained distinctions.
Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs). We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions. Using this rubric, we release a dataset of two experts' annotations for 100 LLM-generated legal propositions. Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones. We further examine LLMs as evaluators and find that rubric-guided LLM judgments align more closely with expert assessments than direct overall scoring, but remain insensitive to finer-grained distinctions captured by human experts.