CitaLaw: Enhancing LLM with Citations in Legal Domain
This work addresses the need for reliable legal AI systems by providing a benchmark for legal citation evaluation, though it is incremental as it builds on existing LLM and retrieval methods.
The authors tackled the problem of evaluating LLMs' ability to produce legally sound responses with citations by introducing CitaLaw, a benchmark with legal questions and a reference corpus, and found that integrating legal references substantially enhances response quality, with their syllogism-based evaluation method showing strong agreement with human judgments.
In this paper, we propose CitaLaw, the first benchmark designed to evaluate LLMs' ability to produce legally sound responses with appropriate citations. CitaLaw features a diverse set of legal questions for both laypersons and practitioners, paired with a comprehensive corpus of law articles and precedent cases as a reference pool. This framework enables LLM-based systems to retrieve supporting citations from the reference corpus and align these citations with the corresponding sentences in their responses. Moreover, we introduce syllogism-inspired evaluation methods to assess the legal alignment between retrieved references and LLM-generated responses, as well as their consistency with user questions. Extensive experiments on 2 open-domain and 7 legal-specific LLMs demonstrate that integrating legal references substantially enhances response quality. Furthermore, our proposed syllogism-based evaluation method exhibits strong agreement with human judgments.