Validity Assessment of Legal Will Statements as Natural Language Inference
This work addresses a domain-specific problem for legal AI by creating a new dataset, but it is incremental as it applies existing NLI methods to a new legal context.
The authors tackled the problem of assessing the validity of statements in legal wills using natural language inference, by introducing a unique dataset with longer texts and three inputs per decision, and found that neural models achieve over 80% macro F1 and accuracy but show superficial understanding with group accuracy in the mid-80s.
This work introduces a natural language inference (NLI) dataset that focuses on the validity of statements in legal wills. This dataset is unique because: (a) each entailment decision requires three inputs: the statement from the will, the law, and the conditions that hold at the time of the testator's death; and (b) the included texts are longer than the ones in current NLI datasets. We trained eight neural NLI models in this dataset. All the models achieve more than 80% macro F1 and accuracy, which indicates that neural approaches can handle this task reasonably well. However, group accuracy, a stricter evaluation measure that is calculated with a group of positive and negative examples generated from the same statement as a unit, is in mid 80s at best, which suggests that the models' understanding of the task remains superficial. Further ablative analyses and explanation experiments indicate that all three text segments are used for prediction, but some decisions rely on semantically irrelevant tokens. This indicates that overfitting on these longer texts likely happens, and that additional research is required for this task to be solved.