LRAGE: Legal Retrieval Augmented Generation Evaluation Tool
This work addresses the need for evaluation tools in legal RAG systems, which is important for researchers and practitioners in legal AI, but it is incremental as it builds on existing RAG evaluation practices.
The authors tackled the problem of evaluating retrieval-augmented generation (RAG) systems in the legal domain by proposing LRAGE, an open-source tool that provides GUI and CLI interfaces for holistic evaluation, and validated it using multilingual legal benchmarks including Korean, English, and Chinese datasets to show how changes in components affect overall accuracy.
Recently, building retrieval-augmented generation (RAG) systems to enhance the capability of large language models (LLMs) has become a common practice. Especially in the legal domain, previous judicial decisions play a significant role under the doctrine of stare decisis which emphasizes the importance of making decisions based on (retrieved) prior documents. However, the overall performance of RAG system depends on many components: (1) retrieval corpora, (2) retrieval algorithms, (3) rerankers, (4) LLM backbones, and (5) evaluation metrics. Here we propose LRAGE, an open-source tool for holistic evaluation of RAG systems focusing on the legal domain. LRAGE provides GUI and CLI interfaces to facilitate seamless experiments and investigate how changes in the aforementioned five components affect the overall accuracy. We validated LRAGE using multilingual legal benches including Korean (KBL), English (LegalBench), and Chinese (LawBench) by demonstrating how the overall accuracy changes when varying the five components mentioned above. The source code is available at https://github.com/hoorangyee/LRAGE.