JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences
This work addresses the need for accurate evaluation in legal NLP applications, benefiting legal practitioners and lay users, though it is incremental as it builds on existing text simplification evaluation methods.
The paper tackles the problem of evaluating legal meaning preservation in text simplification by introducing JUDGEBERT, a new metric that shows superior correlation with human judgment and passes key sanity checks, achieving 100% for identical sentences and 0% for unrelated ones.
Simplifying text while preserving its meaning is a complex yet essential task, especially in sensitive domain applications like legal texts. When applied to a specialized field, like the legal domain, preservation differs significantly from its role in regular texts. This paper introduces FrJUDGE, a new dataset to assess legal meaning preservation between two legal texts. It also introduces JUDGEBERT, a novel evaluation metric designed to assess legal meaning preservation in French legal text simplification. JUDGEBERT demonstrates a superior correlation with human judgment compared to existing metrics. It also passes two crucial sanity checks, while other metrics did not: For two identical sentences, it always returns a score of 100%; on the other hand, it returns 0% for two unrelated sentences. Our findings highlight its potential to transform legal NLP applications, ensuring accuracy and accessibility for text simplification for legal practitioners and lay users.