Chain of Logic: Rule-Based Reasoning with Large Language Models
This addresses the problem of enhancing legal reasoning accuracy for AI applications, though it is incremental as it builds on existing prompting techniques.
The paper tackled the challenge of rule-based reasoning with compositional rules in legal contexts by introducing the Chain of Logic prompting method, which improved performance over other methods like chain of thought and self-ask across eight tasks in the LegalBench benchmark.
Rule-based reasoning, a fundamental type of legal reasoning, enables us to draw conclusions by accurately applying a rule to a set of facts. We explore causal language models as rule-based reasoners, specifically with respect to compositional rules - rules consisting of multiple elements which form a complex logical expression. Reasoning about compositional rules is challenging because it requires multiple reasoning steps, and attending to the logical relationships between elements. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to resolve the underlying logical expression). This method was inspired by the IRAC (Issue, Rule, Application, Conclusion) framework, a sequential reasoning approach used by lawyers. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark and demonstrate it consistently outperforms other prompting methods, including chain of thought and self-ask, using open-source and commercial language models.