Logical Reasoning with Outcome Reward Models for Test-Time Scaling
This work addresses the under-explored area of test-time scaling for logical reasoning in LLMs, offering incremental improvements for AI systems requiring robust deductive capabilities.
The paper tackled the problem of enhancing large language models' performance in deductive logical reasoning by developing Outcome Reward Models (ORMs) trained on Chain-of-Thought and echo-augmented data, resulting in improved performance on datasets like FOLIO, JustLogic, and ProverQA across four LLMs.
Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs' tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.