PersonaMatrix: A Recipe for Persona-Aware Evaluation of Legal Summarization
This work addresses the need for tailored evaluation in legal AI summarization to improve access for both expert and non-expert users, though it is incremental in focusing on evaluation rather than summarization methods.
The paper tackles the problem of evaluating legal document summarization by introducing PersonaMatrix, a persona-aware evaluation framework that scores summaries for six different user personas, and shows divergent optima between persona-aware and persona-agnostic judges using a new dataset and Diversity-Coverage Index.
Legal documents are often long, dense, and difficult to comprehend, not only for laypeople but also for legal experts. While automated document summarization has great potential to improve access to legal knowledge, prevailing task-based evaluators overlook divergent user and stakeholder needs. Tool development is needed to encompass the technicality of a case summary for a litigator yet be accessible for a self-help public researching for their lawsuit. We introduce PersonaMatrix, a persona-by-criterion evaluation framework that scores summaries through the lens of six personas, including legal and non-legal users. We also introduce a controlled dimension-shifted pilot dataset of U.S. civil rights case summaries that varies along depth, accessibility, and procedural detail as well as Diversity-Coverage Index (DCI) to expose divergent optima of legal summary between persona-aware and persona-agnostic judges. This work enables refinement of legal AI summarization systems for both expert and non-expert users, with the potential to increase access to legal knowledge. The code base and data are publicly available in GitHub.