Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
This addresses the challenge of reliably evaluating LLMs on complex, multi-document legal tasks for researchers and practitioners, but it is incremental as it builds on existing evaluation methods with new frameworks and tools.
The paper tackles the problem of evaluating LLMs on long-context legal summarization by introducing Gavel-Ref, a reference-based framework with multi-value checklist evaluation, and finds that even top models like Gemini 2.5 Pro achieve only around 50% on their metric, highlighting task difficulty. It also develops Gavel-Agent, an autonomous agent that reduces token usage by 36% with only a 7% drop in checklist score compared to end-to-end extraction.
Large language models (LLMs) now support contexts of up to 1M tokens, but their effectiveness on complex long-context tasks remains unclear. In this paper, we study multi-document legal case summarization, where a single case often spans many documents totaling 100K-500K tokens. We introduce Gavel-Ref, a reference-based evaluation framework with multi-value checklist evaluation over 26 items, as well as residual fact and writing-style evaluations. Using Gavel-Ref, we go beyond the single aggregate scores reported in prior work and systematically evaluate 12 frontier LLMs on 100 legal cases ranging from 32K to 512K tokens, primarily from 2025. Our results show that even the strongest model, Gemini 2.5 Pro, achieves only around 50 of $S_{\text{Gavel-Ref}}$, highlighting the difficulty of the task. Models perform well on simple checklist items (e.g., filing date) but struggle on multi-value or rare ones such as settlements and monitor reports. As LLMs continue to improve and may surpass human-written summaries -- making human references less reliable -- we develop Gavel-Agent, an efficient and autonomous agent scaffold that equips LLMs with six tools to navigate and extract checklists directly from case documents. With Qwen3, Gavel-Agent reduces token usage by 36% while resulting in only a 7% drop in $S_{\text{checklist}}$ compared to end-to-end extraction with GPT-4.1.