LegalAgentBench: Evaluating LLM Agents in Legal Domain
This addresses the problem of inadequate evaluation tools for LLM agents in legal applications, particularly for researchers and developers, but it is incremental as it builds on existing benchmarking approaches by adding domain-specific features.
The authors tackled the lack of specialized benchmarks for evaluating LLM agents in legal contexts by introducing LegalAgentBench, a comprehensive benchmark for the Chinese legal domain, which includes 17 corpora, 37 tools, and 300 annotated tasks, and they evaluated eight popular LLMs to identify strengths and limitations.
With the increasing intelligence and autonomy of LLM agents, their potential applications in the legal domain are becoming increasingly apparent. However, existing general-domain benchmarks cannot fully capture the complexity and subtle nuances of real-world judicial cognition and decision-making. Therefore, we propose LegalAgentBench, a comprehensive benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain. LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge. We designed a scalable task construction framework and carefully annotated 300 tasks. These tasks span various types, including multi-hop reasoning and writing, and range across different difficulty levels, effectively reflecting the complexity of real-world legal scenarios. Moreover, beyond evaluating final success, LegalAgentBench incorporates keyword analysis during intermediate processes to calculate progress rates, enabling more fine-grained evaluation. We evaluated eight popular LLMs, highlighting the strengths, limitations, and potential areas for improvement of existing models and methods. LegalAgentBench sets a new benchmark for the practical application of LLMs in the legal domain, with its code and data available at \url{https://github.com/CSHaitao/LegalAgentBench}.