Factoring Statutory Reasoning as Language Understanding Challenges
This work addresses statutory reasoning for legal AI applications, but it is incremental as it builds on an existing benchmark with added annotations and structure.
The authors tackled the problem of statutory reasoning by decomposing it into four language-understanding subtasks, using concepts from Prolog programs, and showed that models benefit from this structure, improving on prior baselines.
Statutory reasoning is the task of determining whether a legal statute, stated in natural language, applies to the text description of a case. Prior work introduced a resource that approached statutory reasoning as a monolithic textual entailment problem, with neural baselines performing nearly at-chance. To address this challenge, we decompose statutory reasoning into four types of language-understanding challenge problems, through the introduction of concepts and structure found in Prolog programs. Augmenting an existing benchmark, we provide annotations for the four tasks, and baselines for three of them. Models for statutory reasoning are shown to benefit from the additional structure, improving on prior baselines. Further, the decomposition into subtasks facilitates finer-grained model diagnostics and clearer incremental progress.