Using Language Models For Knowledge Acquisition in Natural Language Reasoning Problems
This addresses the challenge of improving accuracy in AI reasoning tasks for applications like puzzle-solving, but it is incremental as it builds on existing LLM and theorem prover techniques.
The paper tackled the problem of solving natural language reasoning tasks by comparing two approaches: direct LLM solving versus extracting facts for a theorem prover, finding that the latter method is more effective based on experiments with ChatGPT and GPT4 on logic word puzzles.
For a natural language problem that requires some non-trivial reasoning to solve, there are at least two ways to do it using a large language model (LLM). One is to ask it to solve it directly. The other is to use it to extract the facts from the problem text and then use a theorem prover to solve it. In this note, we compare the two methods using ChatGPT and GPT4 on a series of logic word puzzles, and conclude that the latter is the right approach.