Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
This work addresses the problem of ensuring credibility and explainability in AI-generated legal reasoning for legal professionals and researchers, representing an incremental advance by applying existing evaluation methods to a new domain-specific dataset.
The paper tackled the challenge of evaluating LLM-generated reasoning traces in legal domains by introducing LEGIT, a large-scale dataset of 24K instances that converts court judgments into hierarchical trees for assessing issue coverage and correctness, showing that LLMs' legal reasoning is significantly impacted by these factors and that RAG and RL methods offer complementary improvements.
Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs' legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage.