A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences
This work addresses the problem of bias and lack of transparency in AI-assisted law reasoning for legal professionals and the public, though it is incremental as it builds on existing legal applications.
The paper tackles the lack of exploration in law reasoning for AI by proposing a transparent schema with hierarchical structures and creating the first crowd-sourced dataset for a challenging task that outputs justifications for legal decisions, enabling comprehensive evaluation.
While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored. We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation. Simultaneously, we propose an agent framework that employs a comprehensive suite of legal analysis tools to address the challenge task. This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the ``Intelligent Court''.