Large Language Models Imitate Logical Reasoning, but at what Cost?
This addresses the computational efficiency problem for AI researchers and practitioners by offering a more cost-effective method for logical reasoning tasks, though it is incremental as it builds on existing neuro-symbolic approaches.
The study evaluated the reasoning capability of large language models over 18 months, finding that hidden Chain of Thought prompting improved performance from 2023 to 2024, and thinking models led to significant gains from 2024 to 2025, while a neuro-symbolic architecture reduced computational cost by maintaining near-perfect performance with models under 15 billion parameters.
We present a longitudinal study which evaluates the reasoning capability of frontier Large Language Models over an eighteen month period. We measured the accuracy of three leading models from December 2023, September 2024 and June 2025 on true or false questions from the PrOntoQA dataset and their faithfulness to reasoning strategies provided through in-context learning. The improvement in performance from 2023 to 2024 can be attributed to hidden Chain of Thought prompting. The introduction of thinking models allowed for significant improvement in model performance between 2024 and 2025. We then present a neuro-symbolic architecture which uses LLMs of less than 15 billion parameters to translate the problems into a standardised form. We then parse the standardised forms of the problems into a program to be solved by Z3, an SMT solver, to determine the satisfiability of the query. We report the number of prompt and completion tokens as well as the computational cost in FLOPs for open source models. The neuro-symbolic approach significantly reduces the computational cost while maintaining near perfect performance. The common approximation that the number of inference FLOPs is double the product of the active parameters and total tokens was accurate within 10\% for all experiments.