LegalRikai: Open Benchmark -- Benchmark for Complex Japanese Corporate Legal Tasks
This addresses the need for practice-oriented benchmarks in the legal domain, particularly for Japanese corporate law, though it is incremental as it builds on existing evaluation methods.
The paper introduces LegalRikai: Open Benchmark, a new benchmark with four complex tasks based on Japanese corporate legal practices, created by legal professionals and evaluated with 100 samples using human and automated methods on leading LLMs. Results show that automated evaluation aligns well with human judgment for clear criteria but struggles with structural consistency, demonstrating its utility as a screening tool when experts are scarce.
This paper introduces LegalRikai: Open Benchmark, a new benchmark comprising four complex tasks that emulate Japanese corporate legal practices. The benchmark was created by legal professionals under the supervision of an attorney. This benchmark has 100 samples that require long-form, structured outputs, and we evaluated them against multiple practical criteria. We conducted both human and automated evaluations using leading LLMs, including GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1. Our human evaluation revealed that abstract instructions prompted unnecessary modifications, highlighting model weaknesses in document-level editing that were missed by conventional short-text tasks. Furthermore, our analysis reveals that automated evaluation aligns well with human judgment on criteria with clear linguistic grounding, and assessing structural consistency remains a challenge. The result demonstrates the utility of automated evaluation as a screening tool when expert availability is limited. We propose a dataset evaluation framework to promote more practice-oriented research in the legal domain.