Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
This addresses the issue of unreliable logical reasoning in AI systems for applications requiring precise inference, though it is an incremental improvement by combining existing methods.
The paper tackles the problem of large language models struggling with complex logical reasoning by introducing Logic-LM, a framework that integrates LLMs with symbolic solvers, resulting in an average performance boost of 39.2% over LLMs alone with standard prompting and 18.4% over chain-of-thought prompting on five logical reasoning datasets.
Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.